ubuntu_qrt_apparmor will hang with Bionic kernel on KVM / Azure instance (mmap test timeout)

This change was made by a bot.

Hi Po-Hsu,

Alas, I'm not able to reproduce this in a kvm guest with only 768k:

Ran 56 tests in 1026.130s

OK
ubuntu@sec-bionic-amd64:~/tests/qrt-test-apparmor$ free -m
              total used free shared buff/cache available
Mem: 733 543 59 0 130 84
Swap: 947 65 882
ubuntu@sec-bionic-amd64:~/tests/qrt-test-apparmor$ cat /proc/version_signature
Ubuntu 4.15.0-29.31-generic 4.15.18

Host kernel is also running 4.15.0-29.31-generic.

Is it possible one of the prior tests is running longer? Do the kernel-team's tests still build the entire kernel and modules before starting on the tests?

Hi Steve,

The node was deployed by our MaaS, we don't build the kernel before testing.

This was tested manually in two ways:
A. with the autotest framework:

  git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest-client-tests -b master-next
  git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest
  rm -fr autotest/client/tests
  ln -sf ~/autotest-client-tests autotest/client/tests
  AUTOTEST_PATH=/home/ubuntu/autotest sudo -E autotest/client/autotest-local --verbose autotest/client/tests/ubuntu_qrt_apparmor/control

B. Run the python script directlｙ

  # Enable deb-src
  sudo apt-get install -y apparmor apparmor-profiles apparmor-utils apport attr devscripts execstack exim4 gawk git libapparmor-dev libapparmor-perl libcap2-bin libcap-dev libdbus-1-dev libgtk2.0-dev libpam-apparmor netcat pyflakes python3 python3-all-dev python-libapparmor python-pexpect quilt sudo gcc
  git clone --depth 1 https://git.launchpad.net/qa-regression-testing
  cd qa-regression-testing/scripts
  sudo ./test-apparmor.py

I tried several times on this and another KVM　instance, it's 100% reproducible for a fresh node with either method A or B (it looks like the test will pass on second run, not sure if it has anything to do with the test setup process)

Steps:
  1. Run the test either with A or B
  2. Reboot

Everytime when this test hang, it will stuck at the mmap test with operation="profile_load"

This hang (timeout) issue can be found on Azure kernel as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1763002

The zombie process is affecting the automation.

It looks like this hang issue was introduced because the mmap test has timed out.
And it gets killed by the autotest framework.

However it's not really "killed", you can see that it's still stuck with mmap test:

# ps aux
root 23787 0.0 0.0 4384 748 ? S 06:21 0:00 /tmp/testlibpIXHkr/source/bionic/apparmor-2.12/tests/regression/apparmor/mmap

# ps aux |grep Z
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ubuntu 2138 0.0 0.0 0 0 ? Z 05:48 0:00 [sh] <defunct>
root 8307 0.0 0.0 0 0 ? Z 10:15 0:00 [unix_socket_cli] <defunct>

In this case, with PID 23787 killed, the test will keep going and terminates on jenkins properly.

In this cycle, issue spotted on the following Azure nodes with 4.15 kernel:
* Standard_B4ms
* Standard_D4_v3
* Standard_DS1
* Standard_DS12_v2
* Standard_DS13
* Standard_DS13_v2
* Standard_DS14_v2
* Standard_E32-8s_v3
* Standard_GS5-8

For Azure 4.15 kernel on Xenial:
* Standard_B1s
* Standard_B4ms
* Standard_DS12_v2
* Standard_DS14_v2
* Standard_DS14_v2_promo
* Standard_DS1_v2
* Standard_E4_v3
* Standard_D32s_v3

For Azure 4.15 kernel on Trusty:
 * Standard-B1ms
 * Standard_E2s_v3
 * Standard-F1s
 * Standard_F32s_v2

3:15:40 ERROR| [stderr] Run parser regression tests from 14.04's apparmor_2.8.95~2430-0ubuntu5.3 ... (Applying patch r2541-tests-adjust-for-parser-escape-fixes.patch) (Applying patch r2719-parser-tests.patch) (Applying patch r3219-make-sysctl-test-more-resiliant.patch) (Applying patch r3376-parser-tests.patch) (Applying patch r3401-backport-tests-adjust-expected-errno-for-change-profile.patch) (Applying patch r3488-3489-fix_onexec_test_races.patch) (Applying patch tests-convert_ptrace_getregset.patch) (Applying patch tests-readdir-lp1674245.patch) ok
23:15:40 ERROR| [stderr] test_old_trusty_regression_testsuite (__main__.ApparmorTestsuites)
23:15:47 DEBUG| [stdout] preparing apparmor_2.8.95~2430-0ubuntu5.3.dsc...
02:04:07 INFO | Timer expired (10800 sec.), nuking pid 19224
02:04:08 INFO | ERROR ubuntu_qrt_apparmor.test-apparmor.py ubuntu_qrt_apparmor.test-apparmor.py timestamp=1570673048 localtime=Oct 10 02:04:08 Test timeout expired, rc=15
02:04:08 INFO | END ERROR ubuntu_qrt_apparmor.test-apparmor.py ubuntu_qrt_apparmor.test-apparmor.py timestamp=1570673048 localtime=Oct 10 02:04:08

BTW,
as it's been a while, I am not sure if this is still hanging with the mmap test.