xserver mouse pointer emulation from touch breaks with QML app.

Status changed to 'Confirmed' because the bug affects multiple users.

can you reproduce on another touch screen? It seems to work fine on my XPS12

happens on the XPS12 too once you click _into_ an app

some more observations:
- once the bug is activated (by clicking into the touch app), all touch screen events seem to be either discarded (when not in the app surface) or consumed (when in app surface)
- closing the app releases the events and the touch screen works fine again

also, happens with a non-SDK app, e.g. from http://qt-project.org/doc/qt-5/qml-tutorial2.html

import QtQuick 2.0

Item {
    id: container
    property alias cellColor: rectangle.color
    signal clicked(color cellColor)

    width: 40; height: 25

    Rectangle {
        id: rectangle
        border.color: "white"
        anchors.fill: parent
    }

    MouseArea {
        anchors.fill: parent
        onClicked: container.clicked(container.cellColor)
    }
}

On a side note: multitouch in xorg is very problematic. Just now I had to reboot my xps touch screen laptop because the mouse pointer emulation logic no longer worked (pointer simply disappeared on touch). In another attempt playing with multitouch, xserver simply crashed and I was sent back to the login screen.

So, when you run a QML app and tap on it, it seems that the mouse pointer emulation dies but multitouch events still flow normally as unity still responds to 3-finger (window drags, "alt-tab" switch) and 4-finger (show/hide dash, launcher drag) gestures

hmm, and pointer emulation actually still works for other apps when their windows are focused. You just can't focus them by tapping them.

The problem does not occur in a Gnome Flashback (Compiz) session, so it is evidently a result of the interaction between the XInput2 touch grabs used in Unity 7 (through Geis), the passive Xinput touch subscriptions in Qt, and the pointer emulation in x.org.

Multi-touch pointer emulation in XInput2 is documented as acting the way as described in this bug when multi-touch sequences are neither accepted nor rejected (ie pointer emulation stops when the first touch ends - see http://who-t.blogspot.ca/2011/12/multitouch-in-x-pointer-emulation.html) although it is unclear why the emulation does not resume when all touches have stopped.

It sounds like the Unity touch support needs to add gesture accept/reject.

Here is a test setup:

(1) install ubuntu-clock app
(2) modify /usr/share/gnome-session/sessions/ubuntu.session to remove the 'compiz;' entry so compiz gets started by upstart instead of gnome-session
(3) modify /usr/share/upstart/session/unity7.conf to add the following lines in the pre-start script stanza:
  initctl set-env --global GEIS_DEBUG=3
  initctl set-env --global GRAIL_DEBUG=-1
(4) log in to Unity, start the clock app and touch it anywhere in the window

The .cache/upstart/unity7.log contains the following.

[CLOCK] ClockPage loaded
[CLOCK] Saved Current Location: Merrickville
[CLOCK] Saved Current Location Coordinates: -75.8333,44.9167
[CLOCK] Alarm Database loaded
[CLOCK] MainView loaded
[CLOCK] Retrieving Sunrise/Sunset times from disk
GEIS(debug)-geis_backend_multiplexor_pump:414 activity 0x1 on fd 18 callback_info=0x1e62f50
GEIS(debug)-geis_backend_multiplexor_pump:414 activity 0x1 on fd 19 callback_info=0x1e655e0
GRAIL DEBUG (regular-recognizer.cpp:ProcessFrameEvent:54): new event 0x4997c80 with time 161447105
GRAIL DEBUG (regular-recognizer.cpp:ProcessTouches:95): touch 1 began with start time 161447105
GRAIL DEBUG (regular-recognizer.cpp:ProcessTouches:98): touch 1 has been added to all_touches_
GRAIL DEBUG (regular-recognizer.cpp:ProcessTouches:99): touch 1 has been added to free_touches_
GRAIL DEBUG (regular-recognizer.cpp:ProcessFrameEvent:54): new event 0x42edc80 with time 161447108
GRAIL DEBUG (touch.cpp:Update:65): touch 1 is owned
GEIS(debug)-geis_backend_multiplexor_pump:414 activity 0x1 on fd 18 callback_info=0x1e62f50
GRAIL DEBUG (handle.cpp:UpdateTime:197): client updating time to 161447165
GRAIL DEBUG (recognizer.cpp:UpdateTime:304): Updating time to 161447165
GRAIL DEBUG (recognizer.cpp:RejectOverdueGesturesAndTouches:288): touch 1 has been removed from free_touches_ because it is older than the gesture composition time (time: 161447165, touch start time: 161447105)
GRAIL DEBUG (recognizer.cpp:RejectOverdueGesturesAndTouches:292): touch 1 has been erased from free_touches_
GRAIL DEBUG (touch.cpp:~Touch:80): rejecting touch 1
GRAIL DEBUG (recognizer.cpp:UpdateTime:304): Updating time to 161447165
GEIS(warning)-_x11_fd_callback:1160 failed to get X generic event data
GEIS(warning)-_x11_fd_callback:1160 failed to get X generic event data
GEIS(debug)-geis_backend_multiplexor_pump:414 activity 0x1 on fd 19 callback_info=0x1e655e0
GRAIL DEBUG (regular-recognizer.cpp:ProcessFrameEvent:54): new event 0x42edc80 with time 161447166
GRAIL DEBUG (regular-recognizer.cpp:ProcessTouches:126): touch 1 has been erased from all_touches_

From this log you can see that the single touch is reported to Unity and gets rejected because additional touches do not begin within the gesture setup period.

According to the XInput2 documentation, this should forward the TouchBegin to the clock, which should then do with it what it will. The clock should not be receiving emulated pointer events, because according to the XInput2 documentation, if Touch events are explicitly selected, pointer emulation is disabled for that client.

It looks like either (a) the X server is in pointer emulation mode and never leaves, or (b) the client is doing ...

Any QML application at all will do (such as "Item {width:600;height:600}")

What seems to be happening, from what I've learned so far:

- user touches on a qml application
- xserver sends the resulting touch events to both the qmlscene and to compiz/unity7
- xserver offers ownership to compiz/unity7
- qmlscene does XIGrabDevice on the touch screen
- compiz/unity7 rejects the touch
- user lifts finger and touch ends
- qmlscene does XIUngrabDevice on the touch screen
- user lands a new finger, this time on the desktop
- xserver still has in its bookkeeping that the previous touch is active and thus deny pointer emulation for that new touch

And from that point onwards xserver is left with a bogus state, even after the qml app is closed. Restarting it being the sure way to clean it up and reproduce the issue anew (otherwise further issues seem to crop up)

@Daniel, have you tried adding some qDebug() lines in the QT5 libraries to see if we actually get the touch events and finally ungrab the device? (the code is in src/plugins/platforms/xcb/qxcbconnection_xi2.cpp)

Also, it would be interesting to see what XIGrabDevice and XIUngrabDevice return.

dandrader: That seems useful for making a testcase, thanks!

One thing that smells fishy is that the qml app starts getting touch events even before compiz/unity7. I think that this should not happen as the qml app does not subscribe for ownership events (unlike compiz/unity7) and ownership is offered to compiz/unity7.

So the qml app should only get touch events once compiz/unity7 rejects ownership of the touch sequence, but that's just not happening (qml app was already receiving events from that touch sequence by that time).

On my system the touchscreen keeps working after using a QML app. It still gets gestures (e.g 3 fingers on the screen) but it won't work on some windows. Furthermore, I can't drag windows by the decoration any more.

> On my system the touchscreen keeps working after using a QML app. It still gets gestures (e.g 3 fingers on the screen) but it won't work on some windows. Furthermore, I can't drag windows by the decoration any more.

Multifinger gestures keep working because they do not depend on pointer/mouse events, which is what gets broken here (namely "mouse pointer emulation from touch events"). They use touch events directly.

Took the liberty of correcting the bug title as unity does keep getting touch events. It's the xserver mouse pointer emulation from touch events that breaks badly.

Attaching detailed xserver log. The following steps were taken:

1- sudo restart lightdm (to restart the X server. so that de have a clean slate)
2- run a very simple qml app with qmlscene
3- tapped on that qmlscene window (touch id 1)
4- tapped on the desktop area (nautilus) (touch id 2)
5- sudo stop lightdm

From the logs you can see that no pointer emulation was done for touch id 2 because xserver still thinks that touch id 1 is active.

xserver packages I used to take the logs above.

Excerpt from #xorg-devel:

"""
<dandrader> daniels, I've the following situation: clientX has a XISelectEvents(TouchBegin|Update|End, childWindowA, slaveDevice1); clientY has a XIPassiveGrabDevice(TouchBegin|Update|End|Ownership, rootWindow, AllMasterDevices)
 daniels, when a touch reaches xserver, it sends touch events to both clientX and clientY
<daniels> dandrader: yes, events are delivered separately thorough master and slaves - i'd recommend not using slaves for anything at all
<dandrader> daniels, but I believe it should send only to clientY, and only once clientY rejects the touch sequence should clientX finally start receiving the touch events (replayed), right?
<daniels> dandrader: if both clients were listening on a master device, then you'd be right
<dandrader> daniels, hmm got it. so what I'm seeing is the expected behavior then and I should modify clientX to select events on AllMasterDevices instead of on slaveDevice1
 daniels, is that it?
<daniels> dandrader: exactly
"""

So we have two problems here:
  1- touch ownership mechanics won't work in our situation as Qt is talking directly to the slave devices whereas unity7 is talking to the master devices.
   2- the xserver bug described in my previous comments where it gets confused with such situation and ends up with a borked internal state which ruins mouse pointer emulation

So if we could avoid the situation in item 1 (by changing either Qt or unity7/"Open Input Fw Grail" code) we might (have to try it out to see) circumvent the bug in item 2.

Is there a specific reason for calling XISelectEvents() for each slave vs using XIAllMasterDevices in QXcbConnection::xi2Select() in Qt?

 > Is there a specific reason for calling XISelectEvents() for each slave vs using XIAllMasterDevices in QXcbConnection::xi2Select() in Qt?

Would have to dig into Qt code to tell that. But my guess is that because it exposes QTouchDevice objects to applications. For every QTouchEvent you can get the QTouchDevice from where it came from. But it might be that it could work with XIAllMasterDevices just as well.

The code in Qt that subscribes to slave device events is the explicit tablet-handling code. Checking the code, the touchscreen on my Dell XT2 is being treated as a tablet by the Qt code because it supports a stylus mode on one of its alternate devices.

It sounds like the bug is in the Qt XCB QPA code, which explains why it only started occuring in QML apps after the switch to Qt 5.2.

I think the XIDeviceEvent.sourceid would tell you from which device it originates. The XIDeviceEvent.deviceid member would be set to the master device, which might break things in a straight conversion.

@Stephen: I guess that means that your device passes the following check:

if (tabletData.valuatorInfo.contains(QXcbAtom::AbsX) &&
     tabletData.valuatorInfo.contains(QXcbAtom::AbsY) &&
     tabletData.valuatorInfo.contains(QXcbAtom::AbsPressure))

which you can check using xinput list-props $device_id or simply using evtest.

I have a non-tablet and a tablet, they are both affected by the same problem, except that fixing the tablet use case will probably take a little more work.

I've just rebuilt Qt with a partial fix which only covers the "non-tablet" test case, and I'm glad to report that it solves the problem.

Maarten is working on an all comprehensive fix.

And here's the upstream bug I reported:
https://bugs.freedesktop.org/show_bug.cgi?id=78345

@Daniel: thanks for taking the time to file an upstream bug report.

I have tested Maarten's latest work on the Qt library, and I can confirm that it solves the issue on both tablets and non-tablets. Maarten's work is available here (you only need to update the qt packages):
https://launchpad.net/~canonical-x/+archive/x-staging

Status changed to 'Confirmed' because the bug affects multiple users.

I confirm that the Qt from https://launchpad.net/~canonical-x/+archive/x-staging does XISelectEvents on the master pointer (instead of the touchscreen slave device, as previously) and that the resulting behavior of the xserver is good. Ie., touch ownership mechanics work properly and mouse pointer emluation from touch events keeps working as well.

Qt still XISelectEvents my "N-Trig Pen stylus" and "N-Trig Pen eraser" slave devices though. But thankfully no events come from them and thus peace is maintained :)

This bug was fixed in the package qtbase-opensource-src - 5.2.1+dfsg-1ubuntu17

---------------
qtbase-opensource-src (5.2.1+dfsg-1ubuntu17) utopic; urgency=medium

  * Disable PCH also on arm64 in addition to armel/armhf (LP: #1318635)
 -- Timo Jyrinki <email address hidden> Tue, 13 May 2014 04:37:13 +0000

qtbase 5.2.1+dfsg-1ubuntu14.2 SRU for trusty is in unapproved queue now too (https://launchpad.net/ubuntu/trusty/+queue?queue_state=1&queue_text=)

Status changed to 'Confirmed' because the bug affects multiple users.

Hello Steve, or anyone else affected,

Accepted qtbase-opensource-src into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/qtbase-opensource-src/5.2.1+dfsg-1ubuntu14.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I can no longer reproduce this with qtbase 5.2.1+dfsg-1ubuntu14.2 - nice work!

The verification of the Stable Release Update for qtbase-opensource-src has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package qtbase-opensource-src - 5.2.1+dfsg-1ubuntu14.2

---------------
qtbase-opensource-src (5.2.1+dfsg-1ubuntu14.2) trusty; urgency=medium

  * debian/patches/xi2-use-master-device.patch:
    - Make xi2 select events on master device instead of slaves.
      (LP: #1307701)

qtbase-opensource-src (5.2.1+dfsg-1ubuntu14.1) trusty; urgency=medium

  [ Dmitry Shachnev ]
  * Backport upstream patch to fix issues with keymap update handling
    (Add_better_support_for_keymap_update_handling.patch, copied from
    Debian and rebased). (LP: #1318482)
  * Build-depend on libxkbcommon-x11-dev, as the new patch includes
    <xkbcommon/xkbcommon-x11.h>.
 -- Maarten Lankhorst <email address hidden> Tue, 06 May 2014 13:14:17 +0000

When this fixed in 14.04 Trusty Tahr for 14.04.1 (24 July)?

Many "critical" bugs on ReleaseNotes Trusty Tahr page don't fixed presently: https://wiki.ubuntu.com/TrustyTahr/ReleaseNotes#Known_issues
Installation bugs too:
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1066480
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1172572
https://bugs.launchpad.net/ubuntu/+source/graphite2/+bug/1303516
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1297851
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1066342
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1172161
https://bugs.launchpad.net/ubuntu/+source/console-setup/+bug/1297234

upgrade:
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1308530
https://bugs.launchpad.net/ubuntu/+source/tex-common/+bug/1304972
https://bugs.launchpad.net/ubuntu/+source/flightgear/+bug/1308338

and other:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1054732
https://bugs.launchpad.net/unity/+bug/1305586
https://bugs.launchpad.net/ubuntu/+source/gnome-keyring/+bug/1271591
https://bugs.launchpad.net/ubuntu/+source/unity/+bug/1308037
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1305522
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1308761
https://bugs.launchpad.net/ubuntu/+source/unity-greeter/+bug/1292467

Too many errors in Trusty on: http://errors.ubuntu.com
Please fix the most annoying bugs. It interferes with work, use and distribution advice to friends.