Compiz r3275 breaks VirtualBox Guest Additions: black screen or just wallpaper

This indicates a broken driver to me, as the spec says you're not supposed to do that:

#5 0x00007ffff6d22a41 in XOpenDisplay () from /usr/lib/x86_64-linux-gnu/libX11.so.6
No symbol table info available.
#6 0x00007fffea05fc62 in vboxstub_glXBindTexImageEXT () from /usr/lib/x86_64-linux-gnu/dri/vboxvideo_dri.so
No symbol table info available.

---

http://developer.download.nvidia.com/opengl/specs/GLX_EXT_texture_from_pixmap.txt

5. Should users be required to re-bind the drawable to a texture after
    the drawable has been rendered to?

    It is difficult to define what the contents of the texture would be if
    we don't require this. Also, requiring this would allow implementations
    to perform an implicit copy at this point if they could not support
    texturing directly out of renderable memory.

    The problem with defining the contents of the texture after rendering
    has occured to the associated drawable is that there is no way to
    synchronize the use of the buffer as a source and as a destination.
    Direct OpenGL rendering is not necessarily done in the same command
    stream as X rendering. At the time the pixmap is used as the source
    for a texturing operation, it could be in a state halfway through a
    copyarea operation in which half of it is say, white, and half is the
    result of the copyarea operation. How is this defined? Worse, some
    other OpenGL application could be halfway through a frame of rendering
    when the composite manager sources from it. The buffer might just
    contain the results of a "glClear" operation at that point.

    To gurantee tear-free rendering, a composite manager would run as follows:

    -receive request for compositing:
    XGrabServer()
    glXWaitX() or XSync()
    glXBindTexImageEXT()

    <Do rendering/compositing>

    glXReleaseTexImageEXT()
    XUngrabServer()

In any case we'll have to work around it, I don't want to revert this revision as it will cause regressions on nvidia.

Okay,

I had a quick look into the driver and this is what is happening:

 -> glxBindTexImageEXT is bound to DECLEXPORT(void) VBOXGLXTAG(glXBindTexImageEXT)(Display *dpy, GLXDrawable draw, int buffer, const int *attrib_list)
 -> If a pGlxPixmap isn't found it calls stubInitGlxPixmap, which then calls stubInitXDamageExtension
 -> This opens a new X connection in the same process, which deadlocks.

The driver should never be opening a new X connection, but there's a comment there which suggests that they have to do it, possibly because they were working around some broken client's bugs.

"/* Open second xserver connection to make sure we'd receive all the xdamage messages
     * and those wouldn't be eaten by application even queue */
    pContext->damageDpy = XOpenDisplay(DisplayString(pContext->dpy));"

So I think there are two ways around this:

 1) Fix virtualbox so that it does this during glXCreateContext (also broken, but slightly less so)
 2) Work around it in compiz by not doing the server grab for the first pixmap we bind

I have no idea if this patch on virtualbox will even work or compile, but it should work from a 5 minute gloss over the source.

It causes the XDamage extension and subsequent call to XOpenDisplay to happen in glXCreateContext and not glXBindTexImageEXT

I've marked the compiz bug invalid

The attachment "Do xdamage init a bit earlier" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

Thanks for the VirtualBox driver fix Sam! I will try testing this in a moment, compiling VB additions from source. But anyway, it seems to be a bug that probably should be reported upstream, don't you think?

Status changed to 'Confirmed' because the bug affects multiple users.

Bug forwarded upstream.

Workaround committed into lp:compiz at revision 3392