ubuntu_lttng_smoke_test failed on Yakkety s390x (zVM)

Note, this is not a regression, it can be reproduced with 4.8.0-38-generic as well.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1671063

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

sudo strace -f lttng list --kernel --syscall

....
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/lttng/client-lttng-sessiond"}, 110) = 0
geteuid() = 0
getegid() = 0
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=13156}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=8563, uid=0, gid=0}}], msg_controllen=28, msg_flags=0}, 0) = 13156
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\0\0\0\4\0\0\0\32\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=20}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 20
shutdown(3, SHUT_RDWR) = 0
close(3) = 0
write(2, "Error: Unable to list system cal"..., 54Error: Unable to list system calls: Not enough memory
) = 54
geteuid() = 0
getuid() = 0
getegid() = 0
getgid() = 0

This seems wrong - there are no ENOMEM syscall returns to trigger this failure.

https://bugs.lttng.org/issues/1091

end of sudo lttng-sessiond -vvv shows that we have a vmalloc failure:

DEBUG1 - 14:25:37.954060 [10037/10081]: Woken up but nothing in the UST command queue (in thread_dispatch_ust_registration() at main.c:1891)
DEBUG1 - 14:25:37.954072 [10037/10082]: Got the wait shm fd 33 (in get_wait_shm() at shm.c:115)
DEBUG1 - 14:25:37.954074 [10037/10086]: Updating kernel poll set (in update_kernel_poll() at main.c:889)
DEBUG1 - 14:25:37.954100 [10037/10086]: Thread kernel polling (in thread_manage_kernel() at main.c:1099)
DEBUG1 - 14:25:37.953832 [10037/10087]: [load-session-thread] Load session (in thread_load_session() at load-session-thread.c:91)
DEBUG1 - 14:25:37.954079 [10037/10082]: Futex wait update active 1 (in futex_wait_update() at futex.c:66)
DEBUG1 - 14:25:37.954132 [10037/10082]: Accepting application registration (in thread_registration_apps() at main.c:2145)
DEBUG3 - 14:25:37.953889 [10037/10084]: [ust-thread] Manage notify polling (in ust_thread_manage_notify() at ust-thread.c:69)
DEBUG1 - 14:25:42.577842 [10037/10080]: Wait for client response (in thread_manage_clients() at main.c:4444)
DEBUG1 - 14:25:42.577865 [10037/10080]: Receiving data from client ... (in thread_manage_clients() at main.c:4489)
DEBUG1 - 14:25:42.577999 [10037/10080]: Processing client command 4 (in process_client_msg() at main.c:2980)
DEBUG1 - 14:25:42.578010 [10037/10080]: Syscall table listing. (in syscall_table_list() at syscall.c:280)
PERROR - 14:25:42.578021 [10037/10080]: syscall table list zmalloc: Cannot allocate memory (in syscall_table_list() at syscall.c:291)
DEBUG3 - 14:25:42.578027 [10037/10080]: Destroying syscall hash table. (in destroy_syscall_ht() at syscall.c:159)
DEBUG1 - 14:25:42.578030 [10037/10080]: Missing llm structure. Allocating one. (in process_client_msg() at main.c:4124)
DEBUG1 - 14:25:42.578032 [10037/10080]: Sending response (size: 20, retcode: Not enough memory (26)) (in thread_manage_clients()

Hi Colin,

As you indicated it seems that an actual allocation problem occurred on:

syscall table list zmalloc: Cannot allocate memory (in syscall_table_list() at syscall.c:291)
From the code:

/*
 * Allocate at least the number of total syscall we have even if some of
 * them might not be valid. The count below will make sure to return the
 * right size of the events array.
 */
events = zmalloc(syscall_table_nb_entry * sizeof(*events));
if (!events) {
        PERROR("syscall table list zmalloc");
        ret = -LTTNG_ERR_NOMEM;
        goto error;
}
syscall_table_nb_entry is assigned in syscall_init_table.

Could you link the full sessiond log and strace log? Also if possible determine the value of syscall_table_nb_entry?

Cheers

Hi Colin,

Please take a look at the following patch https://lists.lttng.org/pipermail/lttng-dev/2017-March/026959.html

Could you report to us if it fixes your issue?

Please note that s390 is not supported/tested by LTTng-modules.

You probably encountered this bug since no syscalls tracepoints are getting defined on lttng-modules side. Please take a look at this git subtree [1] and the following readme [2] on how to generate it and add the headers for the S390 architecture if you wish to.

Cheers

[1] https://github.com/lttng/lttng-modules/tree/master/instrumentation/syscalls
[2] https://github.com/lttng/lttng-modules/blob/master/instrumentation/syscalls/README

Thanks Jonathan, I'll get around to that first thing tomorrow morning. Much appreciated.

Note: this fixes the issue for me, that is, I don't the vmalloc issues now.

Strangely this didn't get marked Fix Released by the janitor.

ltt-control (2.9.3-1ubuntu1) zesty; urgency=medium

  * Fix lttng list --kernel --syscall failure (LP: #1671063)

 -- Colin Ian King <email address hidden> Tue, 14 Mar 2017 13:39:11 +0000

Hello Po-Hsu, or anyone else affected,

Accepted ltt-control into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ltt-control/2.8.1-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Oh, it didn't get changed because this has a linux not ltt-control bug task.

Hello Po-Hsu, or anyone else affected,

Accepted ltt-control into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ltt-control/2.7.1-2ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tested on s390x xenial and yakkety against -proposed lttng tools in universe - issue fixed.

This bug was fixed in the package ltt-control - 2.7.1-2ubuntu1

---------------
ltt-control (2.7.1-2ubuntu1) xenial; urgency=medium

  * Fix lttng list --kernel --syscall failure (LP: #1671063)

 -- Colin Ian King <email address hidden> Tue, 14 Mar 2017 13:39:11 +0000

The verification of the Stable Release Update for ltt-control has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package ltt-control - 2.8.1-1ubuntu1

---------------
ltt-control (2.8.1-1ubuntu1) yakkety; urgency=medium

  * Fix lttng list --kernel --syscall failure (LP: #1671063)

 -- Colin Ian King <email address hidden> Tue, 14 Mar 2017 13:39:11 +0000