Xwayland crash in gdm as screen off and touchscreen detach.

Bug #1948894 reported by Yuan-Chen Cheng
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mutter
Fix Released
Unknown
OEM Priority Project
Fix Released
Critical
Yuan-Chen Cheng
mutter (Ubuntu)
Fix Released
High
Yuan-Chen Cheng
Focal
Fix Released
High
Yuan-Chen Cheng
Hirsute
Fix Released
High
Yuan-Chen Cheng
Impish
Fix Released
High
Yuan-Chen Cheng

Bug Description

[Impact]

 * Gdm crashed and user need to be an expert to know how to work around,
   or need to reboot to fix.

 * OEM projects got certain SKU that can reliably reproduce this issue.

 * A merged patch in upstream fix the race condition.

[Test Plan]

 * Use the SKU that can reliably reproduce this issue.
 * Do a cold boot, and wait for 5 mins. The monitor should turned-black
   but showing the Machine Vendor logo that was previously
   displayed by plymouth.
 * Try to dog-food the deb from the proposed channel and make sure
   all input events still properly propagate.

[Where problems could occur]

 * If the patch is not properly working as expected, per the file this
   patch touch, the user desktop session could stop responding to
   mouse, keyboard, and touch screen input.

[Other Info]

 * Can't think of any for now.

----

This is a reproducible issue. Step to reproduce
1. cold boot into gdm login screen.
2. wait for 5 mins, and the screen will be turned off. (also observed that the touchscreen USB device drop of a USB bus on the target machine)

Note: this bug seems to be timing sensitive. I got an OEM project machine that can reproduce this issue (reproducible on several machines with the same HW design). The machine is using ADL CPU, which needs a patched 5.14 kernel. So I can't reproduce this on impish. With the backported mutter, I verified an upstream patch seems to fix this issue.

Expected result:
The screen just turns black.

Actual result:
Saw Vendor boot logo on the monitor (previous draw by plymouth during booting)

Analysis as it happens:
1. Saw "LGDisplay Incell Touch" drop off the USB bus.
   Kernel message like: "usb 1-9: USB disconnect, device number 6"
   as the screen turned black.
2. There is no switch on virtual tty.
3. journal log from Xwayland "wl_display@1: error 0: invalid object 24"
4. Xwayland and gnome-shell process both die as this happens.
5. Try to add sleep(30) to OsVendorFatalError(), then both Xwayland and
   gnome-shell stop there for 30 more seconds.
6. backtrace as Xwayland crashed:

```
#0 0x00007f447b8e818b in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f447b8c7859 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x0000564374403f30 in OsAbort () at ../../../../os/utils.c:1351
#3 0x0000564374409369 in AbortServer () at ../../../../os/log.c:872
#4 0x000056437440a1ca in FatalError (f=f@entry=0x564374416266 "%s") at ../../../../os/log.c:1010
#5 0x000056437429c8b0 in xwl_log_handler (format=<optimized out>, args=<optimized out>) at ../../../../../hw/xwayland/xwayland.c:1312
#6 0x00007f447c3d027e in wl_log (fmt=fmt@entry=0x7f447c3d121a "%s@%u: error %d: %s\n") at ../src/wayland-util.c:404
#7 0x00007f447c3cb78b in display_handle_error (data=<optimized out>, display=0x56437639f000, object=0x56437639f000, code=0, message=<optimized out>)
    at ../src/wayland-client.c:911
#8 0x00007f447b7c0ff5 in ?? () from /lib/x86_64-linux-gnu/libffi.so.7
#9 0x00007f447b7c040a in ?? () from /lib/x86_64-linux-gnu/libffi.so.7
#10 0x00007f447c3cf3a8 in wl_closure_invoke (closure=closure@entry=0x564376e57350, flags=flags@entry=1, target=<optimized out>,
    target@entry=0x56437639f000, opcode=opcode@entry=0, data=<optimized out>) at ../src/connection.c:1018
#11 0x00007f447c3cbc48 in dispatch_event (display=display@entry=0x56437639f000, queue=<optimized out>) at ../src/wayland-client.c:1445
#12 0x00007f447c3cd1ef in dispatch_queue (queue=0x56437639f0d0, display=0x56437639f000) at ../src/wayland-client.c:1584
#13 wl_display_dispatch_queue_pending (display=0x56437639f000, queue=0x56437639f0d0) at ../src/wayland-client.c:1833
#14 0x00007f447c3cd280 in wl_display_dispatch_pending (display=<optimized out>) at ../src/wayland-client.c:1896
#15 0x000056437429c9bb in xwl_read_events (xwl_screen=0x564376399dd0) at ../../../../../hw/xwayland/xwayland.c:957
#16 0x0000564374401c61 in ospoll_wait (ospoll=0x56437638f4a0, timeout=<optimized out>) at ../../../../os/ospoll.c:657
#17 0x00005643743faa23 in WaitForSomething (are_ready=0) at ../../../../os/WaitFor.c:208
#18 0x00005643743ca4a7 in Dispatch () at ../../../../include/list.h:220
#19 0x00005643743ce794 in dix_main (argc=16, argv=0x7ffc44d49298, envp=<optimized out>) at ../../../../dix/main.c:276
#20 0x00007f447b8c90b3 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x000056437429c60e in _start ()```

Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :
information type: Proprietary → Public
tags: added: oem-priority originate-from-1947739 somerville
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

This was from focal so Xwayland comes from the 'xorg-server' source package.

In hirsute and later it changed to the 'xwayland' source package.

affects: mutter (Ubuntu) → xorg-server (Ubuntu)
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Oops. That link suggests it's a Wayland server bug in mutter. Fixed in tag 41.0

affects: xorg-server (Ubuntu) → mutter (Ubuntu)
Changed in mutter (Ubuntu):
status: New → Fix Committed
tags: added: fixed-in-41.0 fixed-upstream
tags: added: focal
Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :

Currently we are doing a little bit more test to make sure we this patch does work.

For now, the versions of mutter:
 jammy: 40.5-1ubuntu2
 impish: 40.5-1ubuntu2
Debian:
 sid: 41.0-5

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Expect it to still fail in 40.5-1ubuntu2 as the expected fix (mutter!77) is only in version 41.0.

Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote (last edit ):

debdiff for jammy.

also uploaded to ppa: https://launchpad.net/~ycheng-twn/+archive/ubuntu/mutter-f3

After installing jammy, do some quick trial use of desktop environment with the deb in the ppa works fine so far.

Note: we'd need to SRU back to focal to fulfill the OEM project requirement.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "mutter_40.5-1ubuntu3.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :
description: updated
description: updated
Changed in mutter (Ubuntu):
assignee: nobody → Yuan-Chen Cheng (ycheng-twn)
status: Fix Committed → In Progress
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

In future debdiffs, don't forget to add:

Bug-Ubuntu: https://launchpad.net/bugs/1948894

Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :

debdiff for hirsute. Note I back-port a chunk from the patch.

Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :

add Bug-Ubuntu in salsa git, and re-upload jammy debdiff.

Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :

debdiff for impish

Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote (last edit ):

debdiff for focal. Note I back-port a chunk from the patch as hirsute.

Mathew Hodson (mhodson)
Changed in mutter (Ubuntu):
importance: Undecided → High
affects: mutter (Fedora) → ubuntu
Changed in ubuntu:
importance: Unknown → Undecided
status: Unknown → New
no longer affects: ubuntu
Mathew Hodson (mhodson)
affects: xorg-server → ubuntu
Changed in ubuntu:
importance: Unknown → Undecided
status: Unknown → New
no longer affects: ubuntu
description: updated
Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :
Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote :

Given ubuntu/impish is not exist in salsa, so I create it in my salsa git and another patch to include the change.

https://salsa.debian.org/ycheng-guest/mutter/-/tree/salsa_wayland_race_fix_impish

Changed in mutter (Ubuntu Impish):
assignee: nobody → Yuan-Chen Cheng (ycheng-twn)
importance: Undecided → High
status: New → In Progress
Changed in mutter (Ubuntu Focal):
assignee: nobody → Yuan-Chen Cheng (ycheng-twn)
importance: Undecided → High
status: New → In Progress
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Yuan-Chen, or anyone else affected,

Accepted mutter into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mutter/40.5-1ubuntu3~21.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in mutter (Ubuntu Impish):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-impish
Changed in mutter (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Yuan-Chen, or anyone else affected,

Accepted mutter into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mutter/3.36.9-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Accepted both focal and impish, but I don't see anyone sponsoring the hirsute version - could we get that uploaded as well? I know it's not the latest series, but hirsute will still be supported for a while and we shouldn't forget about users of 21.04, at least not yet. Not everyone upgraded yet.

Can someone sponsor hirsute before we go live with these updates? Thanks.

Revision history for this message
Marco Trevisan (Treviño) (3v1n0) wrote :

Ok, I've uploaded hirsute too, we normally don't have much resources to handle also such versions, but this was easy enough.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Yuan-Chen, or anyone else affected,

Accepted mutter into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mutter/3.38.4-1ubuntu2.21.04.01 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in mutter (Ubuntu Hirsute):
status: New → Fix Committed
tags: added: verification-needed-hirsute
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

From bug 1947739 comment #53:

> test mutter 3.36.0-0ubuntu0.20.04.2 from the focal-proposed channel on
> the target machine for 10 times, can't reproduce this issue.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mutter - 40.5-1ubuntu3

---------------
mutter (40.5-1ubuntu3) jammy; urgency=medium

  [ Yuan-Chen Cheng ]
  * d/p/mutter-wayland-seat-race-fix.patch:
    - Fix Xwayland crash Xwayland crash in gdm as screen off and touchscreen
      detach (cherry picked commits — LP: #1948894)

  [ Marco Trevisan (Treviño) ]
  * debian/patches: Refresh

 -- Marco Trevisan (Treviño) <email address hidden> Wed, 03 Nov 2021 19:37:09 +0100

Changed in mutter (Ubuntu):
status: In Progress → Fix Released
description: updated
Changed in oem-priority:
status: In Progress → Fix Committed
Revision history for this message
Yuan-Chen Cheng (ycheng-twn) wrote (last edit ):

I try to install impish and hirsute on the target machine, with the kernel from focal (so that it support the new ADL GPU, and then the monitor can be turned off so that the touchscreen will be detached, that's the way we reproduce this issue in focal)

With that, with or without mutter from the proposed channel, I can't reproduce this issue. (We do see the backlight turned off and the touchscreen detached from kernel message.)

Given this is a timing-sensitive issue, I think as long as there is no regression, then we can call that as verified done.

I've dog food those, and can't find regression so far. So I'll tag verify done.

tags: added: verification-done verification-done-hirsute verification-done-impish
removed: verification-needed verification-needed-hirsute verification-needed-impish
Mathew Hodson (mhodson)
Changed in mutter (Ubuntu Hirsute):
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mutter - 40.5-1ubuntu3~21.10.1

---------------
mutter (40.5-1ubuntu3~21.10.1) impish; urgency=medium

  * Backport to impish
  * debian: Update vcs references for impish branching

mutter (40.5-1ubuntu3) jammy; urgency=medium

  [ Yuan-Chen Cheng ]
  * d/p/mutter-wayland-seat-race-fix.patch:
    - Fix Xwayland crash Xwayland crash in gdm as screen off and touchscreen
      detach (cherry picked commits — LP: #1948894)

  [ Marco Trevisan (Treviño) ]
  * debian/patches: Refresh

 -- Marco Trevisan (Treviño) <email address hidden> Wed, 03 Nov 2021 19:57:15 +0100

Changed in mutter (Ubuntu Impish):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for mutter has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mutter - 3.38.4-1ubuntu2.21.04.01

---------------
mutter (3.38.4-1ubuntu2.21.04.01) hirsute; urgency=medium

  [ Yuan-Chen Cheng ]
  * d/p/mutter-wayland-seat-race-fix.patch:
    - Fix Xwayland crash Xwayland crash in gdm as screen off and touchscreen
      detach (cherry picked commits — LP: #1948894)
    - backported one chunk of above patch.

  [ Marco Trevisan (Treviño) ]
  * debian: Update VCS references for hirsute branching

 -- Marco Trevisan (Treviño) <email address hidden> Thu, 04 Nov 2021 16:15:12 +0100

Changed in mutter (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mutter - 3.36.9-0ubuntu0.20.04.2

---------------
mutter (3.36.9-0ubuntu0.20.04.2) focal; urgency=medium

  [ Yuan-Chen Cheng ]
  * d/p/mutter-wayland-seat-race-fix.patch:
    - Fix Xwayland crash Xwayland crash in gdm as screen off and touchscreen
      detach (cherry picked commits — LP: #1948894)
    - backported one chunk of above patch.

 -- Marco Trevisan (Treviño) <email address hidden> Wed, 03 Nov 2021 20:10:34 +0100

Changed in mutter (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in mutter (Ubuntu Hirsute):
assignee: nobody → Yuan-Chen Cheng (ycheng-twn)
Changed in oem-priority:
status: Fix Committed → Fix Released
Changed in mutter:
status: Unknown → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.