Firefox break when returning from console with wayland (nvidia) on ubuntu 21.10 beta

Created attachment 9242948
aboutsupport.txt

Gnome Wayland, Debian Testing, Intel

(Darkspirit from bug 1726510 comment 36)
> $ sudo snap remove firefox
> $ sudo snap install firefox
> $ snap run firefox

> WebGL and WebRender work according to about:support, but https://webglsamples.org/aquarium/aquarium.html does not work.
> > It does not appear your computer supports WebGL.
> > Click here for more information.
> > Status: WebGL creation failed: * tryNativeGL (FEATURE_FAILURE_NO_DISPLAY) * Exhausted GL driver options. (FEATURE_FAILURE_WEBGL_EXHAUSTED_DRIVERS)
>
> No word on about:support about this FEATURE_FAILURE_NO_DISPLAY thing.
>
> $ sudo cat /var/log/syslog | grep denied
> > Sep 26 23:36:58 darkspirit-laptop kernel: [41105.417113] audit: type=1400 audit(1632692218.808:14756): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/proc/sys/dev/i915/perf_stream_paranoid" pid=49327 comm="firefox" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
> > Sep 26 23:36:59 darkspirit-laptop kernel: [41105.850276] audit: type=1400 audit(1632692219.240:14757): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/proc/sys/dev/i915/perf_stream_paranoid" pid=49258 comm="Renderer" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

> ---
> Manually installed firefox - without dangerous, but with acking - works the same as official firefox:
> $ sudo snap remove firefox
> $ sudo snap ack firefox_595.assert
> $ sudo snap install firefox_595.snap
> $ snap run firefox

> https://webglsamples.org/aquarium/aquarium.html
> > It does not appear your computer supports WebGL.
> Click here for more information.
> Status: WebGL creation failed: * tryNativeGL (FEATURE_FAILURE_NO_DISPLAY) * Exhausted GL driver options. (FEATURE_FAILURE_WEBGL_EXHAUSTED_DRIVERS)
>
> /var/log/syslog:
> Same two errors as above and this one is new:
> > Sep 26 23:21:31 darkspirit-laptop kernel: [40177.949121] audit: type=1400 audit(1632691291.299:14602): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/proc/46373/environ" pid=46215 comm=427265616B70616420536572766572 requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000

$ snap info firefox
> channels:
> latest/stable: 92.0-3 2021-09-09 (595) 159MB -
> latest/candidate: 92.0.1-1 2021-09-23 (625) 159MB -
> latest/beta: 93.0b9-1 2021-09-24 (628) 155MB -
> latest/edge: ↑
> esr/stable: 78.14.0esr-1 2021-09-07 (591) 148MB -
> esr/candidate: 91.1.0esr-1 2021-09-10 (603) 158MB -
> esr/beta: ↑
> esr/edge: ↑

$ sudo snap remove firefox --purge; sudo snap install firefox --channel=esr/stable; snap run firefox https://webglsamples.org/aquarium/aquarium.html
> firefox (esr/stable) 78.14.0esr-1 from Mozilla✓ installed

works
Edit: because it uses the official Mozilla config (GLX/Xwayland)

$ sudo snap remove firefox; sudo snap install firefox --channel=esr/candidate; snap run firefox https://webglsamples.org/aquarium/aquarium.html
> firefox (esr/candidate) 91.1.0esr-1 from Mozilla✓ installed

broken

$ sudo snap remove firefox; sudo snap install firefox --channel=latest/candidate; snap run firefox https://webglsamples.org/aquarium/aquarium.html
> firefox (candidate) 92.0.1-1 from Mozilla✓ installed

broken

$ sudo snap remove firefox; sudo snap install firefox --channel=latest/beta; snap run firefox https://webglsamples.org/aquarium/aquarium.html
> firefox (beta) 93.0b9-1 from Mozilla✓ installed

broken

93.0b9: Problem does not occur with GLX/Xwayland. Problem also occurs with EGL/Xwayland.
$ DISABLE_WAYLAND=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html

Tested:
* GLX is used on Gnome X11/Nvidia: fine
* EGL is used on Wayland because MOZ_ENABLE_WAYLAND is used (either Snap sets it as default or xwayland is not available)
* As of 94, EGL will also be used on X11 (bug 1695933).

This bug is also reproducible with Firefox Snap 92 when setting gfx.x11-egl.force-enabled to true on Gnome X11/Nvidia, but I don't see any DENIED in syslog.

Hm, this reminds me of bug 1700601 and bug 1696691.

For the record: I'm going to land bug 1732002 to give us some more time to figure out stuff like this. Nvidia does not yet default to Wayland sessions on Ubuntu so setups with default configuration remain unaffected.

Created attachment 9243285
about:support from Firefox snap 92, Gnome X11, Intel

Also reproducible with Firefox Snap 92 and gfx.x11-egl.force-enabled=true on Gnome X11/Intel. No crash report.
Edit: Attachment is XFCE X11, but Gnome X11 looks the same.

glxtest works without problem, WebGL info on about:support is correct. It's just that WebGL doesn't work on websites.

Ah I see. Sounds like it needs to get fixed by whoever maintains the Snap sandbox - likely little we can do in FF.

Pasting a comment of mine from bug 1726510:

« I instrumented the corresponding code and rebuilt the snap: in GetAndInitDisplay() (gfx/gl/GLLibraryEGL.cpp), egl.fGetDisplay() (eglGetDisplay()) always returns null. This requires further investigation. »

The apparmor denials on "/proc/sys/dev/i915/perf_stream_paranoid" mentioned in the description are red herrings: I can see them too, but if I edit the snap's generated apparmor profile (`/var/lib/snapd/apparmor/profiles/snap.firefox.firefox`) to allow read access to that file and reload it, the denials go away but the problem with WebGL persists.

I instrumented further the code, and here are my observations so far:

When launching firefox, the very first call to `GLLibraryEGL::CreateDisplay()` returns a valid EGL display. This is being called from `RenderThread::CreateGLContextEGL()`, from the app's main process.

Subsequent calls to `GLLibraryEGL::CreateDisplay()` follow a different code path (from `GLContextProviderEGL::CreateHeadless()`, from child (content) processes), and those invariably return `null`, which results in WebGL not working.

Running the app with `MOZ_DISABLE_CONTENT_SANDBOX=1` "fixes" WebGL, so there is something in the content sandbox that prevents access to an EGL display.

Disabling widget.dmabuf-webgl.enabled doesn't make a difference.

Summary:
STR on Debian Testing:
$ sudo apt install snapd
$ sudo systemctl start snapd

$ sudo snap install firefox

When using a Wayland session:
* EGL/Wayland backend:
  broken: `$ snap run firefox https://webglsamples.org/aquarium/aquarium.html`
  works: `$ MOZ_DISABLE_CONTENT_SANDBOX=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`
* EGL/Xwayland:
  broken: `$ DISABLE_WAYLAND=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`
  works: `$ MOZ_DISABLE_CONTENT_SANDBOX=1 DISABLE_WAYLAND=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`
* fine with glx/xwayland: `$ DISABLE_WAYLAND=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`

When using an X11 session:
* EGL/X11:
  broken: `$ MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`
  works: `$ MOZ_DISABLE_CONTENT_SANDBOX=1 MOZ_X11_EGL=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html`
* fine with GLX/X11: `$ snap run firefox https://webglsamples.org/aquarium/aquarium.html`

Also, reducing `security.sandbox.content.level` to `2` (from its default value of `4`) in `about:config` makes the problem go away.

And setting `security.sandbox.content.read_path_whitelist` to `$SNAP/` (e.g. `/snap/firefox/595/`, the trailing slash is important), also makes the problem go away, confirming that the content process sandboxing is what prevents the WebGL code from accessing the EGL library.

I'm tempted to patching `SandboxBrokerPolicyFactory::InitContentPolicy()` to call `policy->AddPath(rdonly, "$SNAP/")` when firefox is running as a snap. This will obviously require a thorough security review.

I would expect https://searchfox.org/mozilla-central/source/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp#499 to pick up libraries shipped alongside Firefox. I guess what's happening here is that the library is shipped in the snap (but not in the default system), not next to the binary, and then "something" is done to make the dynamic linker pick it up?

Our sandbox knows about LD_LIBRARY_PATH and such https://searchfox.org/mozilla-central/rev/c3d7964c593e0bedabea2fea0b35ba243cf9e696/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp#258 but I guess this is using something different?

In general readonly permission to trusted system library dirs should not be a security concern.

Indeed, the library is shipped by the snap. To be exact, it is shipped by the platform snap that the firefox snap uses as its base (gnome-3-38-2004), and the snap sees it at `$SNAP/gnome-platform/usr/lib/x86_64-linux-gnu/libEGL.so`. The snap's launcher modifies `LD_LIBRARY_PATH` accordingly. This is the value for a webcontent (child) process (where `x21` is the snap's revision, because I manually installed an instrumented build):

    LD_LIBRARY_PATH=/snap/firefox/x21/usr/lib/firefox:/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/firefox/x21/usr/lib:/snap/firefox/x21/usr/lib/x86_64-linux-gnu:/snap/firefox/x21/gnome-platform/lib/x86_64-linux-gnu:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu:/snap/firefox/x21/gnome-platform/usr/lib:/snap/firefox/x21/gnome-platform/lib:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu/dri:/var/lib/snapd/lib/gl:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu/libunity:/snap/firefox/x21/gnome-platform/usr/lib/x86_64-linux-gnu/pulseaudio

The path in question is there, so it's not immediately clear to me why it's not being added to the policy's list of readonly paths. Maybe the call to `realpath(…)` doesn't work well with the snap's confinement?

I was able to narrow down the one missing path (required to make WebGL work) to `$SNAP/gnome-platform/usr/share/glvnd/egl_vendor.d/`.

That path contains one single file (`50_mesa.json`):

    {
        "file_format_version" : "1.0.0",
        "ICD" : {
            "library_path" : "libEGL_mesa.so.0"
        }
    }

So the problem is not with `LD_LIBRARY_PATH` or with how the sandbox parses its value to add readonly permissions, it is with a configuration file that's located in a place that the policy doesn't allow read access to by default, and which contains information for EGL to locate the right implementation to load.

In that light, and considering that we might see other similar bugs (caused by configuration files for certain libs not being readable) in the future, my suggestion in comment 15 seems to be a valid and future-proof solution.

Created attachment 9244377
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

Pushed by <email address hidden>:
https://hg.mozilla.org/integration/autoland/rev/f52ebe8f3b52
Allow read access to files under $SNAP/ in the webcontent sandbox. r=gcp

https://hg.mozilla.org/mozilla-central/rev/f52ebe8f3b52

The patch landed in nightly and beta is affected.
:olivier, is this bug important enough to require an uplift?
If not please set `status_beta` to `wontfix`.

For more information, please visit [auto_nag documentation](https://wiki.mozilla.org/Release_Management/autonag#uplift_beta.py).

Yes, this definitely needs to be uplifted to beta.

Comment on attachment 9244377
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

### Beta/Release Uplift Approval Request
* **User impact if declined**: WebGL doesn't work for users of the firefox snap in Wayland sessions.
* **Is this code covered by automated tests?**: Yes
* **Has the fix been verified in Nightly?**: Yes
* **Needs manual test from QE?**: No
* **If yes, steps to reproduce**:
* **List of other uplifts needed**: None
* **Risk to taking this patch**: Low
* **Why is the change risky/not risky? (and alternatives if risky)**: Not risky because only the snap package is affected (the sandbox allows one additional folder for readonly access if the $SNAP environment variable is set).
* **String changes made/needed**:

This patch also appears to fix another problem that happens only in Wayland sessions, and was reported in Ubuntu (against the upcoming 21.10 release which ships the Firefox snap by default): https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1946599.

Comment on attachment 9244377
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

Approved for 94.0b5.

https://hg.mozilla.org/releases/mozilla-beta/rev/111441e21e18

Does this need an ESR91 approval request also?

Yes indeed, this is clearly a candidate for an ESR91 uplift. Thanks for the suggestion Ryan.

Comment on attachment 9244377
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

### ESR Uplift Approval Request
* **If this is not a sec:{high,crit} bug, please state case for ESR consideration**: Snap packaging has changed significantly between ESR78 and ESR91. In ESR78 WebGL works in Wayland sessions, whereas it doesn't in ESR91, making it a very visible functional regression.
* **User impact if declined**: WebGL doesn't work for users of the firefox snap in Wayland sessions.
* **Fix Landed on Version**: 94.0b5
* **Risk to taking this patch**: Low
* **Why is the change risky/not risky? (and alternatives if risky)**: Not risky because only the snap package is affected (the sandbox allows one additional folder for readonly access if the $SNAP environment variable is set).
* **String or UUID changes made by this patch**:

Comment on attachment 9244377
Bug 1732580 - Allow read access to files under $SNAP/ in the webcontent sandbox.

Approved for 91.3esr.

Looks like this depends on bug 1718084, though?

Yes, I hadn't realized this, but it depends on bug 1718084 indeed.

If pulling in the patches from bug 1718084 is deemed too risky for ESR91, I can rework this patch to not depend on them.
That would imply no unit tests, but on the upside the patch would be trivial.

I think we should take it unless there's a strong reason I'm missing. It's been baking for a couple months now, we're early in the ESR91 lifespan, and I won't be surprised if we eventually need to backport other patches that depend on it.

https://hg.mozilla.org/releases/mozilla-esr91/rev/13c638762db7

(In reply to Darkspirit from comment #1)
> $ sudo snap remove firefox; sudo snap install firefox --channel=latest/beta; snap run firefox https://webglsamples.org/aquarium/aquarium.html
> > firefox (beta) 93.0b9-1 from Mozilla✓ installed
>
> broken

Debian Testing, Gnome Wayland, Intel

sudo snap remove firefox; sudo snap install firefox --channel=latest/beta

EGL/Wayland works:
snap run firefox https://webglsamples.org/aquarium/aquarium.html

EGL/Xwayland works:
DISABLE_WAYLAND=1 snap run firefox https://webglsamples.org/aquarium/aquarium.html

*** Bug 1732582 has been marked as a duplicate of this bug. ***

*** Bug 1737737 has been marked as a duplicate of this bug. ***

FTR, the `AddFutureDir` we had to land on that patch was finally removed in bug 1750539.