Incompatible jbd2 format between kernel and lttng-modules

Proposed fix is attached.

Tested on a local vm with kernel focal-5.4.0-143.160.
Initially, sudo apt install lttng-modules-dkms failed for obvious reasons.

Manually installed the new deb ~20.0.5:

ubuntu@focal-143:~/lttng$ dpkg -l | grep lttng-modules
ii lttng-modules-dkms 2.12.5-1ubuntu2~20.04.5 all Linux Trace Toolkit (LTTng) kernel modules (DKMS)

The attachment "20.0.5-20.0.4.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

$ dput ubuntu lttng-modules_2.12.5-1ubuntu2~20.04.5_source.changes
D: Setting host argument.
Checking signature on .changes
gpg: /tmp/lttng-modules_2.12.5-1ubuntu2~20.04.5_source.changes: Valid signature from 9B8EC849D5EF70ED
Checking signature on .dsc
gpg: /tmp/lttng-modules_2.12.5-1ubuntu2~20.04.5.dsc: Valid signature from 9B8EC849D5EF70ED
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading lttng-modules_2.12.5-1ubuntu2~20.04.5.dsc: done.
  Uploading lttng-modules_2.12.5-1ubuntu2~20.04.5.debian.tar.xz: done.
  Uploading lttng-modules_2.12.5-1ubuntu2~20.04.5_source.buildinfo: done.
  Uploading lttng-modules_2.12.5-1ubuntu2~20.04.5_source.changes: done.
Successfully uploaded packages.

Sponsored to the unapproved queue.

Hello Roxana, or anyone else affected,

Accepted lttng-modules into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.12.5-1ubuntu2~20.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

 lttng module 2.12.5-1ubuntu2~20.04.5 in -proposed tested on Focal 5.4.0-1094-oracle.

The module can be built and tested without any issue.

 == lttng smoke test of session create/destroy ==
 Session test-kernel-session created.
 Traces will be output to /tmp/lttng-kernel-trace-24770-session
 PASSED (lttng create)
 Session "test-kernel-session" destroyed
 PASSED (lttng destroy)

 == lttng smoke test trace context switches ==
 Session test-kernel-session created.
 Traces will be output to /tmp/lttng-kernel-trace-24770-session
 PASSED (lttng create)
 Kernel event sched_switch created in channel channel0
 PASSED (lttng enable-event)
 Tracing started for session test-kernel-session
 PASSED (lttng start)
 Waiting for data availability.
 Tracing stopped for session test-kernel-session
 PASSED (lttng stop)
 Waiting for destruction of session "test-kernel-session"..
 Session "test-kernel-session" destroyed
 PASSED (lttng destroy)
 Found 129766 dd and 196500 context switches
 PASSED (simple system call tracing with babeltrace)

 Summary: 7 passed, 0 failed

Fix for bionic: https://launchpadlibrarian.net/650753152/18.04.4-5.debdiff

Backport for jammy. Tested autopkgtest locally:
I: Testing if lttng-modules modules are correctly installed
lttng-modules/2.13.8, 5.15.0-66-generic, x86_64: installed
./lttng-modules/2.13.8/5.15.0-57-generic/x86_64/log/make.log
./lttng-modules/2.13.8/5.15.0-66-generic/x86_64/log/make.log
./lttng-modules/2.13.8/5.15.0-58-generic/x86_64/log/make.log
./lttng-modules/2.13.8/5.15.0-59-generic/x86_64/log/make.log
autopkgtest [12:33:34]: test dkms-autopkgtest: -----------------------]
autopkgtest [12:33:35]: test dkms-autopkgtest: - - - - - - - - - - results - - - - - - - - - -
dkms-autopkgtest PASS
autopkgtest [12:33:36]: @@@@@@@@@@@@@@@@@@@@ summary
dkms-autopkgtest PASS

Backport for kinetic. Test results:
I: Testing if lttng-modules modules are correctly installed
lttng-modules/2.13.8, 5.19.0-35-generic, x86_64: installed
./lttng-modules/2.13.8/5.19.0-31-generic/x86_64/log/make.log
./lttng-modules/2.13.8/5.19.0-35-generic/x86_64/log/make.log
./lttng-modules/2.13.8/5.19.0-29-generic/x86_64/log/make.log
I: Summary:
I: PASS 5.19.0-29-generic
I: PASS 5.19.0-31-generic
I: PASS 5.19.0-35-generic
autopkgtest [12:32:19]: test dkms-autopkgtest: -----------------------]
autopkgtest [12:32:20]: test dkms-autopkgtest: - - - - - - - - - - results - - - - - - - - - -
dkms-autopkgtest PASS
autopkgtest [12:32:20]: @@@@@@@@@@@@@@@@@@@@ summary
dkms-autopkgtest PASS

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2004644

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Build verified on Bionic for the debdiff provided above: https://launchpadlibrarian.net/650753152/18.04.4-5.debdiff

$ dput ubuntu lttng-modules_2.10.8-1ubuntu2~18.04.5_source.changes
D: Setting host argument.
Checking signature on .changes
gpg: /tmp/lttng-modules_2.10.8-1ubuntu2~18.04.5_source.changes: Valid signature from 9B8EC849D5EF70ED
Checking signature on .dsc
gpg: /tmp/lttng-modules_2.10.8-1ubuntu2~18.04.5.dsc: Valid signature from 9B8EC849D5EF70ED
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading lttng-modules_2.10.8-1ubuntu2~18.04.5.dsc: done.
  Uploading lttng-modules_2.10.8-1ubuntu2~18.04.5.debian.tar.xz: done.
  Uploading lttng-modules_2.10.8-1ubuntu2~18.04.5_source.buildinfo: done.
  Uploading lttng-modules_2.10.8-1ubuntu2~18.04.5_source.changes: done.
Successfully uploaded packages.

@roxana for the new upstream version backports, the debdiff is normally provided against the version that is being backported e.g. relative 2.13.8-1

Also when doing backport / any ubuntu change, the version string must include ubuntu in it, so something like ~ubuntu22.10.0 is best.

Jammy debdiff

Kinetic debdiff

1) changed -1ubuntu1~22.04.0 to -1~ubuntu22.04.0 because -1ubuntu1~22.04.0 is higher version that is in lunar:

$ dpkg --compare-versions 2.13.8-1ubuntu1~22.04.0 lt 2.13.8-1 && echo "sru version is lower than devel" || echo "sru version is HIGHER than devel"
sru version is HIGHER than devel

$ dpkg --compare-versions 2.13.8-1~ubuntu22.04.0 lt 2.13.8-1 && echo "sru version is lower than devel" || echo "sru version is HIGHER than devel"
sru version is lower than devel

we want to ensure that lunar version remains higher than the sru backport versions

2) run update-maintainer as the -1 version was synced from debian without ubuntu maintainer set

3) generated jammy changes with $ debuild -S -d -nc -v2.13.1-1

similar done for kinetic too.

both sponsored.

Hello Roxana, or anyone else affected,

Accepted lttng-modules into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.13.8-1~ubuntu22.10.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Roxana, or anyone else affected,

Accepted lttng-modules into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lttng-modules/2.13.8-1~ubuntu22.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I came across a similar problem in the following situation. Maybe it can help you move forward quickly. Because it is base on docker and cross compile.

git clone https://github.com/ros-realtime/linux-real-time-kernel-builder.git

$ cd linux-real-time-kernel-builder

$ docker build --no-cache --build-arg UBUNTU_VERSION=focal --build-arg KERNEL_VERSION=5.4.0 --build-arg RT_PATCH=5.4.230-rt80 --build-arg LTTNG_VERSION=2.13 -t rtwg-image .

After that the Docker image is prepared and ready to run
$ docker run -it rtwg-image bash

and then inside the docker
$ cd $HOME/linux_build/linux-raspi
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- LOCALVERSION=-raspi -j `nproc` bindeb-pkg

A error part of screen out;

make KERNELRELEASE=5.4.229-rt80-raspi ARCH=arm64 KBUILD_BUILD_VERSION=1 -f ./Makefile
  CALL scripts/atomic/check-atomics.sh
  CALL scripts/checksyscalls.sh
  CHK include/generated/compile.h
  CC lttng/src/lttng-ring-buffer-client-discard.o
  CC lttng/src/lttng-ring-buffer-client-overwrite.o
  CC lttng/src/probes/lttng-probe-jbd2.o
  CC lttng/src/probes/lttng-probe-scsi.o
  CC lttng/src/probes/lttng-probe-vmscan.o
In file included from lttng/src/probes/../../include/lttng/define_trace.h:87,
                 from lttng/src/probes/../../include/instrumentation/events/jbd2.h:293,
                 from lttng/src/probes/lttng-probe-jbd2.c:29:
lttng/src/probes/../../include/lttng/tracepoint-event-impl.h:133:6: error: conflicting types for ‘trace_jbd2_run_stats’
  133 | void trace_##_name(_proto);
      | ^~~~~~
lttng/src/probes/../../include/lttng/tracepoint-event-impl.h:45:2: note: in expansion of macro ‘LTTNG_TRACEPOINT_EVENT_INSTANCE_MAP’
   45 | LTTNG_TRACEPOINT_EVENT_INSTANCE_MAP(map, name, map, PARAMS(proto), PARAMS(args))
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lttng/src/probes/../../include/lttng/tracepoint-event-impl.h:87:2: note: in expansion of macro ‘LTTNG_TRACEPOINT_EVENT_MAP’
   87 | LTTNG_TRACEPOINT_EVENT_MAP(name, name, \
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~

version 2.13.8-1~ubuntu22.10.0 tested on Kinetic 5.19.0-31, test passed without any issue.
version 2.13.8-1~ubuntu22.04.0 tested on Jammy 5.15.0-60, test passed without any issue

This is also affecting Bionic hwe SRU.

version 2.10.8-1ubuntu2~18.04.5 tested on bionic 5.4.0-144.161~18.04.1 works fine
https://autopkgtest.ubuntu.com/results/autopkgtest-bionic/bionic/amd64/l/lttng-modules/20230223_150053_15e8f@/log.gz

Has lttng-smoke-test been run against the packages from -proposed? It's not *entirely* clear to me, but that looked to be a part of the test-case, and the autopkgtests don't (appear?) to run the smoke test?

Yes, indeed, if you check adt matrix, only build is done. Smoke testing is done during rt. They were tested locally (expect bionic which is already in updates)
1. bionic was tested on linux-gpg-4.15.0-1146.162 in rt:
731 12:56:40 DEBUG| [stdout] Preparing to unpack .../76-lttng-modules-dkms_2.10.8-1ubuntu2~18.04.5_all.deb ...
      732 12:56:40 DEBUG| [stdout] Unpacking lttng-modules-dkms (2.10.8-1ubuntu2~18.04.5) ...
...

      1470 12:58:42 INFO | == lttng smoke test trace context switches ==
      1471 12:58:42 INFO | Session test-kernel-session created.
      1472 12:58:42 INFO | Traces will be written in /tmp/lttng-kernel-trace-22702-session
      1473 12:58:42 INFO | PASSED (lttng create)
      1474 12:58:42 INFO | Kernel event sched_switch created in channel channel0
      1475 12:58:42 INFO | PASSED (lttng enable-event)
      1476 12:58:42 INFO | Tracing started for session test-kernel-session
      1477 12:58:42 INFO | PASSED (lttng start)
      1478 12:58:42 INFO | Waiting for data availability.
      1479 12:58:42 INFO | Tracing stopped for session test-kernel-session
      1480 12:58:42 INFO | PASSED (lttng stop)
      1481 12:58:42 INFO | Session test-kernel-session destroyed
      1482 12:58:42 INFO | PASSED (lttng destroy)
      1483 12:58:42 INFO | Found 26292 dd and 147227 context switches
      1484 12:58:42 INFO | PASSED (simple system call tracing with babeltrace)
      1485 12:58:42 INFO |
      1486 12:58:42 INFO | Summary: 7 passed, 0 failed
      1487 12:58:42 DEBUG| Running 'apt-get remove --yes lttng-modules-dkms'
      1488 12:59:27 INFO | GOOD ubuntu_lttng_smoke_test.lttng-smoke-test ubuntu_lttng_smoke_test.lttng-smoke-test timestamp=1677416367 localtime=Feb 26 12:59:27 completed successfully
      1489 12:59:27 INFO | END GOOD ubuntu_lttng_smoke_test.lttng-smoke-test ubuntu_lttng_smoke_test.lttng-smoke-test timestamp=1677416367 localtime=Feb 26 12:59:27

2. focal 5.4.0-145.162
ubuntu@focal:~$ apt search lttng-modules-dkms
Sorting... Done
Full Text Search... Done
lttng-modules-dkms/focal-proposed 2.12.5-1ubuntu2~20.04.5 all
  Linux Trace Toolkit (LTTng) kernel modules (DKMS)

13:51:57 INFO | Summary: 7 passed, 0 failed
13:51:57 DEBUG| Running 'apt-get remove --yes lttng-modules-dkms'
13:52:21 INFO | GOOD ubuntu_lttng_smoke_test.lttng-smoke-test ubuntuy
13:52:21 INFO | END GOOD ubuntu_lttng_smoke_test.lttng-smoke-test ubuntu1
13:52:21 DEBUG| Persistent state client._record_indent now set to 1
13:52:21 DEBUG| Persistent state client.unexpected_reboot deleted
13:52:21 INFO | END GOOD ---- ---- timestamp=1678197141 localtime=Mar 07 13:521

3. jammy 5.15.0-68.75
ubuntu@jammy:~$ apt search lttng-modules-dkms
Sorting... Done
Full Text Search... Done
lttng-modules-dkms/jammy-proposed,now 2.13.8-1~ubuntu22.04.0 all [installed]
  Linux Trace Toolkit (LTTng) kernel modules (DKMS)

14:16:39 INFO | Summary: 7 passed, 0 failed
14:16:39 DEBUG| Running 'apt-get remove --yes lttng-modules-dkms'
14:16:52 INFO | GOOD ubun...

Hello Chris and Timo. Can you let us know if anything else is needed to push focal, jammy, and kinetic versions in -updates?

This bug was fixed in the package lttng-modules - 2.13.8-1~ubuntu22.10.0

---------------
lttng-modules (2.13.8-1~ubuntu22.10.0) kinetic; urgency=medium

  * Backport dkms support for hwe support (LP: #2004644)

lttng-modules (2.13.8-1) unstable; urgency=medium

  * [813bc03] New upstream version 2.13.8

lttng-modules (2.13.7-1) unstable; urgency=medium

  * [a85260b] New upstream version 2.13.7
  * [8da9d13] Update dkms config (Closes: #1019250)

 -- Roxana Nicolescu <email address hidden> Tue, 14 Feb 2023 15:07:11 +0100

The verification of the Stable Release Update for lttng-modules has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package lttng-modules - 2.13.8-1~ubuntu22.04.0

---------------
lttng-modules (2.13.8-1~ubuntu22.04.0) jammy; urgency=medium

  * Backport dkms support for hwe support (LP: #2004644)

lttng-modules (2.13.8-1) unstable; urgency=medium

  * [813bc03] New upstream version 2.13.8

lttng-modules (2.13.7-1) unstable; urgency=medium

  * [a85260b] New upstream version 2.13.7
  * [8da9d13] Update dkms config (Closes: #1019250)

lttng-modules (2.13.4-1) unstable; urgency=medium

  * [431cdf5] New upstream version 2.13.4

lttng-modules (2.13.3-2) unstable; urgency=medium

  [ Andreas Beckmann ]
  * [1a7054d] switch to dh-sequence-dkms
  * [b5f9942] switch to Testsuite: autopkgtest-pkg-dkms
  * [8a3be1e] look for kernel config in $kernel_source_dir/.config first
  * [eceedfa] the autopkgtests may run on Salsa CI

lttng-modules (2.13.3-1) unstable; urgency=medium

  * [74b0b84] New upstream version 2.13.3

lttng-modules (2.13.2-1) unstable; urgency=medium

  * [dbf1674] New upstream version 2.13.2

 -- Roxana Nicolescu <email address hidden> Tue, 14 Feb 2023 15:09:16 +0100

Hey Brian! Focal version 2.12.5-1ubuntu2~20.04.5 is still pending in -proposed. Is it possible to move it to updates as well?

This bug was fixed in the package lttng-modules - 2.12.5-1ubuntu2~20.04.5

---------------
lttng-modules (2.12.5-1ubuntu2~20.04.5) focal; urgency=medium

  * Backport fixes to jbd2 format in 5.4.229 (LP: #2004644)
    - d/p/0013-fix-jbd2-use-the-correct-print-format.patch
    - d/p/0014-fix-jbd2-use-the-correct-print-format-v5.10.163.patch
    - d/p/0015-fix-jbd2-upper-bound-for-v5.10.163.patch
    - d/p/0016-fix-jbd2-use-the-correct-print-format-v5.4.229.patch

 -- Roxana Nicolescu <email address hidden> Fri, 03 Feb 2023 11:09:19 +0200