T2 Phoenix in Jenkins (Cloudera) creating cores in QueryExec tests
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Trafodion |
Fix Committed
|
High
|
Weiqing Xu |
Bug Description
On CDH and CM distros, Phoenix T2 tests failing during QueryExec* tests and creating cores with the stack trace below.
Thread 24 (Thread 0x7fffecbfb700 (LWP 14543)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75a918b in ?? () from /usr/lib/
#2 0x00007ffff7576e78 in ?? () from /usr/lib/
#3 0x00007ffff75770ff in ?? () from /usr/lib/
#4 0x00007ffff7635394 in ?? () from /usr/lib/
#5 0x00007ffff76dcca8 in ?? () from /usr/lib/
#6 0x00007ffff71b74c0 in ?? () from /usr/lib/
#7 0x00007ffff75c118c in ?? () from /usr/lib/
#8 0x00007ffff72195ae in ?? () from /usr/lib/
#9 0x00007ffff721bd3f in ?? () from /usr/lib/
#10 0x00007ffff718d00c in ?? () from /usr/lib/
#11 0x00007ffff722145b in ?? () from /usr/lib/
#12 0x00007ffff7222ac1 in ?? () from /usr/lib/
#13 0x00007ffff76e0ba5 in ?? () from /usr/lib/
#14 0x00007ffff76e0e30 in ?? () from /usr/lib/
#15 0x00007ffff75af9a2 in ?? () from /usr/lib/
#16 0x00000030aba079d1 in start_thread () from /lib64/
#17 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fffc9e2c700 (LWP 14966)):
#0 0x00000030ab63371d in sigtimedwait () from /lib64/libc.so.6
#1 0x00007fffe707a5e4 in local_monitor_
#2 0x00000030aba079d1 in start_thread () from /lib64/
#3 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fffec1c8700 (LWP 14554)):
#0 0x00000030aba0b98e in pthread_
#1 0x00007ffff75ae457 in ?? () from /usr/lib/
#2 0x00007ffff76feba0 in ?? () from /usr/lib/
#3 0x00007fffed0127f8 in ?? ()
#4 0x00007fffed006058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 21 (Thread 0x7fffecdfd700 (LWP 14541)):
#0 0x00000030aba0d930 in sem_wait () from /lib64/
#1 0x00007ffff75aeaec in ?? () from /usr/lib/
#2 0x00007ffff75a79ba in ?? () from /usr/lib/
#3 0x00007ffff76e0ba5 in ?? () from /usr/lib/
#4 0x00007ffff76e0e30 in ?? () from /usr/lib/
#5 0x00007ffff75af9a2 in ?? () from /usr/lib/
#6 0x00000030aba079d1 in start_thread () from /lib64/
#7 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7ffff5a93700 (LWP 14535)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75a918b in ?? () from /usr/lib/
#2 0x00007ffff75774c3 in ?? () from /usr/lib/
#3 0x00007ffff7577a9e in ?? () from /usr/lib/
#4 0x00007ffff731f0d3 in ?? () from /usr/lib/
#5 0x00007ffff73206f4 in ?? () from /usr/lib/
#6 0x00007ffff75af9a2 in ?? () from /usr/lib/
#7 0x00000030aba079d1 in start_thread () from /lib64/
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fffecefe700 (LWP 14539)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75a918b in ?? () from /usr/lib/
#2 0x00007ffff759ecdc in ?? () from /usr/lib/
#3 0x00007ffff76a2eb8 in ?? () from /usr/lib/
#4 0x00007ffff7419760 in JVM_MonitorWait () from /usr/lib/
#5 0x00007fffed0127f8 in ?? ()
#6 0x0000000000000000 in ?? ()
Thread 18 (Thread 0x7ffff7bdc720 (LWP 14517)):
#0 0x00000030aba0822d in pthread_join () from /lib64/
#1 0x000000344d208ab5 in ?? () from /usr/lib/
#2 0x000000344d202574 in ?? () from /usr/lib/
#3 0x000000344d2050a8 in JLI_Launch () from /usr/lib/
#4 0x0000000000400766 in ?? ()
#5 0x00000030ab61ed5d in __libc_start_main () from /lib64/libc.so.6
#6 0x0000000000400629 in ?? ()
#7 0x00007fffffffe3d8 in ?? ()
#8 0x000000000000001c in ?? ()
#9 0x0000000000000008 in ?? ()
#10 0x00007fffffffe65e in ?? ()
#11 0x00007fffffffe697 in ?? ()
#12 0x00007fffffffe6a2 in ?? ()
#13 0x00007fffffffe6da in ?? ()
#14 0x00007fffffffe711 in ?? ()
#15 0x00007fffffffe736 in ?? ()
#16 0x00007fffffffe768 in ?? ()
#17 0x00007fffffffe76d in ?? ()
#18 0x0000000000000000 in ?? ()
Thread 17 (Thread 0x7ffff6dca700 (LWP 14532)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75a918b in ?? () from /usr/lib/
#2 0x00007ffff75774c3 in ?? () from /usr/lib/
#3 0x00007ffff7577b36 in ?? () from /usr/lib/
#4 0x00007ffff772989c in ?? () from /usr/lib/
#5 0x00007ffff75cc2c1 in ?? () from /usr/lib/
#6 0x00007ffff76f3bae in ?? () from /usr/lib/
#7 0x00007ffff7633cde in ?? () from /usr/lib/
#8 0x00007fffed05f5c7 in ?? ()
#9 0x00000007870a6818 in ?? ()
#10 0x00007fffed1462c0 in ?? ()
#11 0x00007ffff6dc2800 in ?? ()
#12 0x00000007d99fc108 in ?? ()
#13 0x0000003a00000051 in ?? ()
#14 0x00000007870a6838 in ?? ()
#15 0x00000007dbddf7e0 in ?? ()
#16 0x00000007000000ae in ?? ()
#17 0x00000007870a6818 in ?? ()
#18 0x0000000000000001 in ?? ()
#19 0x0000000785000508 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 16 (Thread 0x7ffff5992700 (LWP 14536)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75a918b in ?? () from /usr/lib/
#2 0x00007ffff75774c3 in ?? () from /usr/lib/
#3 0x00007ffff7577a9e in ?? () from /usr/lib/
#4 0x00007ffff731f0d3 in ?? () from /usr/lib/
#5 0x00007ffff73206f4 in ?? () from /usr/lib/
#6 0x00007ffff75af9a2 in ?? () from /usr/lib/
#7 0x00000030aba079d1 in start_thread () from /lib64/
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7ffff5b94700 (LWP 14534)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75a918b in ?? () from /usr/lib/
#2 0x00007ffff75774c3 in ?? () from /usr/lib/
#3 0x00007ffff7577a9e in ?? () from /usr/lib/
#4 0x00007ffff731f0d3 in ?? () from /usr/lib/
#5 0x00007ffff73206f4 in ?? () from /usr/lib/
#6 0x00007ffff75af9a2 in ?? () from /usr/lib/
#7 0x00000030aba079d1 in start_thread () from /lib64/
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fffd4f18700 (LWP 14965)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007fffe4ec8656 in SB_Thread::CV::wait (this=0x7ffff0f
at /home/jenkins/
#2 0x00007fffe4ec8732 in SB_Thread::CV::wait (this=0x7ffff0f
at /home/jenkins/
#3 0x00007fffe70aec67 in SB_Sig_
#4 0x00007fffe70bc997 in SB_Trans:
#5 0x00007fffe70bc25a in sock_helper_
#6 0x00007fffe4ec5b9f in SB_Thread:
#7 0x00007fffe4ec5ff7 in thread_fun (pp_arg=
#8 0x00007fffe4ec9290 in sb_thread_sthr_disp (pp_arg=
#9 0x00000030aba079d1 in start_thread () from /lib64/
#10 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fffb616e700 (LWP 14968)):
#0 0x00000030ab6e8ef3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fffe70b4cf5 in SB_Trans:
at sock.cpp:336
#2 0x00007fffe70b4213 in SB_Trans:
#3 0x00007fffe70b404d in sock_comp_
#4 0x00007fffe4ec5b9f in SB_Thread:
#5 0x00007fffe4ec5ff7 in thread_fun (pp_arg=
#6 0x00007fffe4ec9290 in sb_thread_sthr_disp (pp_arg=
#7 0x00000030aba079d1 in start_thread () from /lib64/
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7ffff5c95700 (LWP 14533)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75a918b in ?? () from /usr/lib/
#2 0x00007ffff75774c3 in ?? () from /usr/lib/
#3 0x00007ffff7577a9e in ?? () from /usr/lib/
#4 0x00007ffff731f0d3 in ?? () from /usr/lib/
#5 0x00007ffff73206f4 in ?? () from /usr/lib/
#6 0x00007ffff75af9a2 in ?? () from /usr/lib/
#7 0x00000030aba079d1 in start_thread () from /lib64/
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fffd5919700 (LWP 14964)):
#0 0x00000030aba0ea5d in accept () from /lib64/
#1 0x00007fffe70b59b5 in SB_Trans:
#2 0x00007fffe70bc506 in SB_Trans:
#3 0x00007fffe70bc233 in sock_stream_
#4 0x00007fffe4ec5b9f in SB_Thread:
#5 0x00007fffe4ec5ff7 in thread_fun (pp_arg=
#6 0x00007fffe4ec9290 in sb_thread_sthr_disp (pp_arg=
#7 0x00000030aba079d1 in start_thread () from /lib64/
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fffc942b700 (LWP 14967)):
#0 0x00000030aba0ef3d in nanosleep () from /lib64/
#1 0x00007fffe6c344cb in Sleep (milliSecs=6000) at traf_misc.cpp:136
#2 0x00007fffe51a1298 in memMonitorUpdat
#3 0x00000030aba079d1 in start_thread () from /lib64/
#4 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fffe7bf7700 (LWP 14975)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75ae59a in ?? () from /usr/lib/
#2 0x00007ffff76feba0 in ?? () from /usr/lib/
#3 0x00007fffed0127f8 in ?? ()
#4 0x00007fffed006058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 8 (Thread 0x7fffe7cf8700 (LWP 14902)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75ae59a in ?? () from /usr/lib/
#2 0x00007ffff76feba0 in ?? () from /usr/lib/
#3 0x00007fffed0127f8 in ?? ()
#4 0x00007fffed006058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7fffe7af6700 (LWP 14974)):
#0 0x00000030ab6e8ef3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fffec6f1b3b in Java_sun_
#2 0x00007fffed0127f8 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7fffec18f700 (LWP 14901)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75ae59a in ?? () from /usr/lib/
#2 0x00007ffff76feba0 in ?? () from /usr/lib/
#3 0x00007fffed0127f8 in ?? ()
#4 0x00007fffed006058 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7fffec9f9700 (LWP 14545)):
#0 0x00000030aba0b98e in pthread_
#1 0x00007ffff75ae8af in ?? () from /usr/lib/
#2 0x00007ffff7577758 in ?? () from /usr/lib/
#3 0x00007ffff7577a9e in ?? () from /usr/lib/
#4 0x00007ffff76da5e3 in ?? () from /usr/lib/
#5 0x00007ffff76dab4f in ?? () from /usr/lib/
#6 0x00007ffff75af9a2 in ?? () from /usr/lib/
#7 0x00000030aba079d1 in start_thread () from /lib64/
#8 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fffecafa700 (LWP 14544)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75a918b in ?? () from /usr/lib/
#2 0x00007ffff75774c3 in ?? () from /usr/lib/
#3 0x00007ffff7577a9e in ?? () from /usr/lib/
#4 0x00007ffff7638589 in ?? () from /usr/lib/
#5 0x00007ffff76e0ba5 in ?? () from /usr/lib/
#6 0x00007ffff76e0e30 in ?? () from /usr/lib/
#7 0x00007ffff75af9a2 in ?? () from /usr/lib/
#8 0x00000030aba079d1 in start_thread () from /lib64/
#9 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fffeccfc700 (LWP 14542)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75a918b in ?? () from /usr/lib/
#2 0x00007ffff7576e78 in ?? () from /usr/lib/
#3 0x00007ffff75770ff in ?? () from /usr/lib/
#4 0x00007ffff7635394 in ?? () from /usr/lib/
#5 0x00007ffff76dcca8 in ?? () from /usr/lib/
#6 0x00007ffff71a9473 in ?? () from /usr/lib/
#7 0x00007ffff71c53e3 in ?? () from /usr/lib/
#8 0x00007ffff75e1865 in ?? () from /usr/lib/
#9 0x00007ffff75df3b4 in ?? () from /usr/lib/
#10 0x00007ffff75d2442 in ?? () from /usr/lib/
#11 0x00007ffff75d34e2 in ?? () from /usr/lib/
#12 0x00007ffff75d607f in ?? () from /usr/lib/
#13 0x00007ffff718dfc8 in ?? () from /usr/lib/
#14 0x00007ffff721bbf4 in ?? () from /usr/lib/
#15 0x00007ffff718d00c in ?? () from /usr/lib/
#16 0x00007ffff722145b in ?? () from /usr/lib/
#17 0x00007ffff7222ac1 in ?? () from /usr/lib/
#18 0x00007ffff76e0ba5 in ?? () from /usr/lib/
#19 0x00007ffff76e0e30 in ?? () from /usr/lib/
#20 0x00007ffff75af9a2 in ?? () from /usr/lib/
#21 0x00000030aba079d1 in start_thread () from /lib64/
#22 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fffecfff700 (LWP 14538)):
#0 0x00000030aba0b5bc in pthread_
#1 0x00007ffff75a918b in ?? () from /usr/lib/
#2 0x00007ffff759ecdc in ?? () from /usr/lib/
#3 0x00007ffff76a2eb8 in ?? () from /usr/lib/
#4 0x00007ffff7419760 in JVM_MonitorWait () from /usr/lib/
#5 0x00007fffed0127f8 in ?? ()
#6 0x00007fffecffea50 in ?? ()
#7 0x00007ffff0075ff0 in ?? ()
#8 0x00007fffecffe880 in ?? ()
#9 0x00007fffecffe818 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7ffff4195700 (LWP 14537)):
#0 0x00000030ab632625 in raise () from /lib64/libc.so.6
#1 0x00000030ab633e05 in abort () from /lib64/libc.so.6
#2 0x00007ffff75ae9e5 in ?? () from /usr/lib/
#3 0x00007ffff772281f in ?? () from /usr/lib/
#4 0x00007ffff75b3e12 in JVM_handle_
#5 <signal handler called>
#6 0x00007ffff7611f52 in ?? () from /usr/lib/
#7 0x00007ffff769f0c6 in ?? () from /usr/lib/
#8 0x00007ffff761125b in ?? () from /usr/lib/
#9 0x00007ffff7611d8b in ?? () from /usr/lib/
#10 0x00007ffff75cbd7d in ?? () from /usr/lib/
#11 0x00007ffff7723e77 in ?? () from /usr/lib/
#12 0x00007ffff772a3f2 in ?? () from /usr/lib/
#13 0x00007ffff7728865 in ?? () from /usr/lib/
#14 0x00007ffff7728d23 in ?? () from /usr/lib/
#15 0x00007ffff77291f2 in ?? () from /usr/lib/
#16 0x00007ffff75af9a2 in ?? () from /usr/lib/
#17 0x00000030aba079d1 in start_thread () from /lib64/
#18 0x00000030ab6e88fd in clone () from /lib64/libc.so.6
Changed in trafodion: | |
milestone: | r1.1 → r1.0.1 |
Changed in trafodion: | |
milestone: | r1.0.1 → r1.1 |
Changed in trafodion: | |
status: | New → In Progress |
Changed in trafodion: | |
assignee: | Arvind Narain (arvind-narain) → Kevin Xu (kai-hua-xu) |
Changed in trafodion: | |
assignee: | Kevin Xu (kai-hua-xu) → Weiqing Xu (wei-qing-xu) |
Changed in trafodion: | |
status: | In Progress → Fix Committed |
Made the following changes to test:
1. Commented out nproc entry in /etc/security/limits.d/90-nproc.conf ( to enable values set in /etc/security/limits.conf to be picked up ) - didn't helpjdk1.7.0_67-cloudera ) rather than what is being used in the test - /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64 - didn't help though the stack gave more information ( attached)
2. Used the default java ( /usr/java/
stack quite similar to what is reported here - https://github.com/ochafik/nativelibs4java/issues/420 /bugs.openjdk.java.net/browse/JDK-8025834?page=com.atlassian.streams.streams-jira-plugin:activity-stream-issue-tab
and here
https:/
#0 0x00000033ff632625 in raise () from /lib64/libc.so.6jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so:report_and_die() ()jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.solinux_signal ()jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.sofalse>::do_oop(oopDesc**) ()jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so:oops_do(OopClosure*) ()jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.soy::oops_do(OopClosure*) ()jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.sosk::do_it(GCTaskManager*, unsigned int) () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.sojdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.sojdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.solibpthread.so.0
#1 0x00000033ff633e05 in abort () from /lib64/libc.so.6
#2 0x00007ffff736fa55 in os::abort(bool) ()
from /usr/java/
#3 0x00007ffff74eff87 in VMError:
from /usr/java/
#4 0x00007ffff737496f in JVM_handle_
from /usr/java/
#5 <signal handler called>
#6 0x00007ffff73d9c5c in PSRootsClosure<
from /usr/java/
#7 0x00007ffff703d6dd in Dictionary:
from /usr/java/
#8 0x00007ffff746fe78 in SystemDictionar
from /usr/java/
#9 0x00007ffff73da321 in ScavengeRootsTa
#10 0x00007ffff70a804f in GCTaskThread::run() ()
from /usr/java/
#11 0x00007ffff7370988 in java_start(Thread*) ()
from /usr/java/
#12 0x00000033ffa079d1 in start_thread () from /lib64/
#13 0x00000033ff6e88fd in clone () from /lib64/libc.so.6
(gdb)
3. Suspecting GC - modified the python script to download the repository for maven before running the tests - didn't help - output attached dependency tree might give a clue.
# clean the targetgvars.my_EXPORT_CMD + ';mvn clean')
output = shell_call(
stdout_write(output + '\n')
# download the dependencies firstgvars.my_EXPORT_CMD + ';mvn dependency:tree -Dverbose')
output = shell_call(
stdout_write(output + '\n')
# download the dependencies firstgvars.my_EXPORT_CMD + ';mvn compile')
output = shell_call(
stdout_write(output + '\n')
# download the dependencies first
output = shell_call('echo $CLASSPATH')
stdout_write(output + '\n')
# do the whole build, including running testsgvars.my_EXPORT_CMD + ';mvn test -Dtest=' + ArgList._tests)
output = shell_call(
====
next step
- try by using the release version rather than debug versiondledOops is setup anywhere
- check if -XX:+CheckUnhan