I had a quick look into the driver and this is what is happening:
-> glxBindTexImageEXT is bound to DECLEXPORT(void) VBOXGLXTAG(glXBindTexImageEXT)(Display *dpy, GLXDrawable draw, int buffer, const int *attrib_list)
-> If a pGlxPixmap isn't found it calls stubInitGlxPixmap, which then calls stubInitXDamageExtension
-> This opens a new X connection in the same process, which deadlocks.
The driver should never be opening a new X connection, but there's a comment there which suggests that they have to do it, possibly because they were working around some broken client's bugs.
"/* Open second xserver connection to make sure we'd receive all the xdamage messages
* and those wouldn't be eaten by application even queue */
pContext->damageDpy = XOpenDisplay(DisplayString(pContext->dpy));"
So I think there are two ways around this:
1) Fix virtualbox so that it does this during glXCreateContext (also broken, but slightly less so)
2) Work around it in compiz by not doing the server grab for the first pixmap we bind
Okay,
I had a quick look into the driver and this is what is happening:
-> glxBindTexImageEXT is bound to DECLEXPORT(void) VBOXGLXTAG( glXBindTexImage EXT)(Display *dpy, GLXDrawable draw, int buffer, const int *attrib_list) Extension
-> If a pGlxPixmap isn't found it calls stubInitGlxPixmap, which then calls stubInitXDamage
-> This opens a new X connection in the same process, which deadlocks.
The driver should never be opening a new X connection, but there's a comment there which suggests that they have to do it, possibly because they were working around some broken client's bugs.
"/* Open second xserver connection to make sure we'd receive all the xdamage messages >damageDpy = XOpenDisplay( DisplayString( pContext- >dpy)); "
* and those wouldn't be eaten by application even queue */
pContext-
So I think there are two ways around this:
1) Fix virtualbox so that it does this during glXCreateContext (also broken, but slightly less so)
2) Work around it in compiz by not doing the server grab for the first pixmap we bind