Running g5g.2xlarge image - iOS WebRTC Camera crash

Bug #1982804 reported by Terence
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Anbox Cloud
Fix Released
High
Gary.Wang

Bug Description

We tried running anbox cloud latest 1.14.2 while using image g5g.2xlarge image

When we start using iPhone safari browser, click on the Camera , Allowing permission, we get a crashed message from Anbox, however, if we are using Chrome OSX Browser or Android Chrome browser, it work perfectly fine. Also When we are using c4g.2xlarge instance there isn't any error for iOS Safari

$ amc logs cbfnopv6ndea9ecsosg0

Jul 26 05:32:51 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Reached target Multi-User System.
Jul 26 05:32:51 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Reached target Graphical Interface.
Jul 26 05:32:51 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Startup finished in 23.151s.
Jul 26 05:32:51 ams-cbfnopv6ndea9ecsosg0 anbox-starter[144]: (webrtc_video_engine.cc:1129): Attempting to get RTP parameters for the default, unsignaled video receive stream, but not yet configured to receive such a stream.
Jul 26 05:32:52 ams-cbfnopv6ndea9ecsosg0 anbox-starter[144]: (webrtc_video_engine.cc:1129): Attempting to get RTP parameters for the default, unsignaled video receive stream, but not yet configured to receive such a stream.
Jul 26 05:32:53 ams-cbfnopv6ndea9ecsosg0 anbox-starter[144]: (webrtc_video_engine.cc:1129): Attempting to get RTP parameters for the default, unsignaled video receive stream, but not yet configured to receive such a stream.
Jul 26 05:32:54 ams-cbfnopv6ndea9ecsosg0 anbox-starter[144]: (webrtc_video_engine.cc:1129): Attempting to get RTP parameters for the default, unsignaled video receive stream, but not yet configured to receive such a stream.
Jul 26 05:32:55 ams-cbfnopv6ndea9ecsosg0 anbox-starter[1907]: [1907:1907:20220726,053255.582570:ERROR elf_dynamic_array_reader.h:64] tag not found
Jul 26 05:32:55 ams-cbfnopv6ndea9ecsosg0 anbox-starter[1907]: [1907:1907:20220726,053255.582673:ERROR elf_dynamic_array_reader.h:64] tag not found
Jul 26 05:32:55 ams-cbfnopv6ndea9ecsosg0 anbox-starter[1907]: [1907:1907:20220726,053255.582754:ERROR elf_dynamic_array_reader.h:64] tag not found
Jul 26 05:32:55 ams-cbfnopv6ndea9ecsosg0 anbox-starter[1907]: [1907:1907:20220726,053255.582835:ERROR elf_dynamic_array_reader.h:64] tag not found
Jul 26 05:32:55 ams-cbfnopv6ndea9ecsosg0 anbox-starter[1907]: [1907:1907:20220726,053255.591449:ERROR file_io_posix.cc:144] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2)
Jul 26 05:32:55 ams-cbfnopv6ndea9ecsosg0 anbox-starter[1907]: [1907:1907:20220726,053255.591499:ERROR file_io_posix.cc:144] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2)
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: anbox.service: Main process exited, code=killed, status=11/SEGV
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 sh[1950]: 2022/07/26 05:32:56 Using deprecated hook: backup
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 amsctl[1957]: 2022/07/26 05:32:56 Dumping system.log ...
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 amsctl[1957]: 2022/07/26 05:32:56 Dumping android.log ...
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 amsctl[1957]: 2022/07/26 05:32:56 Copying crash logs from /var/lib/anbox/crashes/session ...
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 amsctl[1957]: 2022/07/26 05:32:56 Copying crash logs from /var/lib/anbox/data/anr ...
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 amsctl[1957]: 2022/07/26 05:32:56 Copying crash logs from /var/lib/anbox/data/tombstones ...
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 amsctl[1957]: 2022/07/26 05:32:56 Status change notified: error 0 service killed with status SEGV
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: anbox.service: Failed with result 'signal'.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Received SIGRTMIN+3.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Removed slice Slice /system/modprobe.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped target Graphical Interface.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped target Multi-User System.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped target Login Prompts.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped target Network is Online.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped target Network.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped target Preparation for Network.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped target Remote File Systems.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped target Timer Units.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: dpkg-db-backup.timer: Deactivated successfully.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped Daily dpkg database backup timer.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: e2scrub_all.timer: Deactivated successfully.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped Periodic ext4 Online Metadata Check for All Filesystems.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: systemd-tmpfiles-clean.timer: Deactivated successfully.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped Daily Cleanup of Temporary Directories.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopping Anbox Container Manager...
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopping Anbox Network Bridge...
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopping Create final runtime dir for shutdown pivot root...
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd-tmpfiles[2081]: /run/finalrd-libs.conf:8: Duplicate line for path "/run/initramfs/lib", ignoring.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: anbox-network-bridge.service: Deactivated successfully.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped Anbox Network Bridge.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: finalrd.service: Deactivated successfully.
Jul 26 05:32:56 ams-cbfnopv6ndea9ecsosg0 systemd[1]: Stopped Create final runtime dir for shutdown pivot root.
Jul 26 05:33:29 ams-cbfnopv6ndea9ecsosg0 anbox[2084]: Execution of hook stop failed: calling container hook timed out
Jul 26 05:34:00 ams-cbfnopv6ndea9ecsosg0 anbox[114]: Container cannot shutdown orderly, forcibly stopping it now
Jul 26 05:34:00 ams-cbfnopv6ndea9ecsosg0 anbox[114]: EPIPE while sending: write: Broken pipe

-------------------

id: cbfnopv6ndea9ecsosg0
name: ams-cbfnopv6ndea9ecsosg0
status: running
node: lxd0
created_at: 2022-07-26 05:32:23 +0000 UTC
application:
  id: cbfnaqv6ndea9ecsos9g
network:
  address: 192.168.100.2
  public_address: **.**.**.**
  services: []
stored_logs: []
error_message: ""
config:
  platform: webrtc
app_id: cbfnaqv6ndea9ecsos9g
app_version: 0
image_id: ""
image_version: 0

------------------
ubuntu@ip-192-168-3-249:~$ amc show-log cbfnopv6ndea9ecsosg0 system.log
Error: log not found

------------------
ubuntu@ip-192-168-3-249:~$ amc show-log cbfnopv6ndea9ecsosg0 android.log
Error: log not found

Terence (terencetwork)
description: updated
information type: Public → Private
information type: Private → Public
Gary.Wang (gary-wzl77)
Changed in anbox-cloud:
assignee: nobody → Gary.Wang (gary-wzl77)
milestone: none → 1.15.0
Revision history for this message
Gary.Wang (gary-wzl77) wrote :

Hey Terence
  Thanks for the bug report.
  We confirm the issue that when streaming from safari(IOS devices), a native crash happened if opening up the camera application. And this issue only happened to applications that are created with a GPU instance type, like g4.3. That's the reason why you hit this issue when you installed Anbox Cloud appliance on g5g.2xlarge instance(where Nvidia T4G GPU is plugged) but not on c4g.2xlarge.
  I just created an internal bug tracking for this. The fix to this bug should be included in the next minor release(1.15.0) and I will keep you posted on any updates for the upcoming release.

BR
Gary

Revision history for this message
Terence (terencetwork) wrote :

Hi Gary,

I've received,
Anbox stream failed Error: failed to communicate with the signaler

Do you think it's the same issue caused by g4.3?

1) I've tried to delete the session and recreate
2) I've to restart the whole system
3) I've tried to delete/recreate the application

All of it is not working, now i can't login into any of the session.

Would you be able to suggest any logs i should look into?

Revision history for this message
Gary.Wang (gary-wzl77) wrote :

Do you think it's the same issue caused by g4.3?
A: no, that's a different issue.
   Do you mean whenever you create a new session from a dashboard, it failed with an error like
"Anbox stream failed Error: failed to communicate with the signaler"?

   Looks like the dashboard was unable to access the stream gateway. It seemed to me it stopped working all of sudden, am i right? Do you recall what have you done prior to the issue occurrence?
   Are you streaming from safari on ios device or chrome? any chance you can debug ios safari for your mac [1] and check the logs or do a remote debug if you stream from Android Chrome browser

   Would you mind dumpping the logs of anbox-stream-gateay via:
   $ juju ssh -m appliance:anbox-cloud anbox-stream-gateway/0 sudo snap logs -n=all anbox-stream-gateway

[1] https://www.busbud.com/blog/debug-ios-safari-mac/
[2] https://developer.chrome.com/docs/devtools/remote-debugging/

BR
Gary

Revision history for this message
Terence (terencetwork) wrote :

2022-08-02T03:20:48Z anbox-stream-gateway.daemon[3529]: I0802 03:20:48.097555 3529 http_api.go:987] Service: Disconnecting previous client before preparing session cbjjj83hnbs028ptncn0
2022-08-02T03:20:48Z anbox-stream-gateway.daemon[3529]: E0802 03:20:48.097983 3529 http_api.go:1022] Service: Failed to disconnect clients for session cbjjj83hnbs028ptncn0: <nil>

Revision history for this message
Terence (terencetwork) wrote :

I recall trying to use the Javascript SDK on another domain
https://github.com/anbox-cloud/anbox-streaming-sdk/tree/master/js

An while connecting and refreshing, it suddenly not working. And all the session suddenly stopped even new session

Revision history for this message
Gary.Wang (gary-wzl77) wrote :

1. Regarding the logs that you shared above, that's all you have? it seems to me the logs of anbox-stream-gateway got truncated and from the last few lines, all looked fine.
Could you paste full logs of anbox-stream-gateway by running
```
juju ssh -m appliance:anbox-cloud anbox-stream-gateway/0 sudo snap logs -n=all anbox-stream-gateway
```

2. As I mentioned above, you could do a remote debug by connecting client device, and what do you find from the console by following [1] or [2]?

3. If you don't have an Android or IOS device to do the remote debug, what if you navigate to the dashboard from the desktop browser, like chrome, and enable dev tool, can you launch a new session successfully? if not, what kind of error message do you see from the `console` tab ? [3]

BR
Gary

[1] https://www.busbud.com/blog/debug-ios-safari-mac/
[2] https://developer.chrome.com/docs/devtools/remote-debugging/
[3] https://developer.chrome.com/docs/devtools/console/

Revision history for this message
Terence (terencetwork) wrote :

1. That's the error message i've received, it wasn't truncated

2. Sadly i destroyed the instance and it couldn't be replicated

3. I was using Mac OS Chrome browser (Desktop)
 - Launched a new session successfully? Yes, amc ls showed instance running and dashboard appeared green
 - Console tab - i've launched through dashboard

Revision history for this message
Gary.Wang (gary-wzl77) wrote :

2. Sadly i destroyed the instance and it couldn't be replicated
A: yeah, that's a bit sad.

3. I was using Mac OS Chrome browser (Desktop)
 - Launched a new session successfully? Yes, amc ls showed instance running and dashboard appeared green
A:And you can get stream working when launching a new session from the dashboard ? when you click the `Exit stream` and Join the same session, is everything working fine?

 - Console tab - i've launched through dashboard
A: Any error you find until now whenever you launch a new session or join a session?

Revision history for this message
Terence (terencetwork) wrote :

Can get stream working when launching a new session from the dashboard ?
- No - Anbox stream failed Error: failed to communicate with the signaler

When you click the `Exit stream` and Join the same session, is everything working fine?
- No - Anbox stream failed Error: failed to communicate with the signaler

Any error you find until now whenever you launch a new session or join a session?
- No error when i see the screen, the instance run successfully, waiting for the session to load... but no screen appearing, when refresh - Anbox stream failed Error: failed to communicate with the signaler

Simon Fels (morphis)
Changed in anbox-cloud:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Terence (terencetwork) wrote :

I've replicated the issue

1) Setup Cloud Application Init
2) Setup Application without APK on Dashboard
3) Run a session
- Work all fine
5) Bind a domain name with the instance IP address example.com -> 1.2.3.4 (instance ip)
4) Setup domain name TLS using Certbot on example.com
https://discourse.ubuntu.com/t/set-up-tls-for-the-anbox-cloud-appliance/28552
5) Setup streaming client token
https://anbox-cloud.io/docs/howto/stream/access
5) Setup JS SDK running the example js
https://github.com/anbox-cloud/anbox-streaming-sdk/tree/master/examples/js
- Replace the gateway with https://example.com
- Replacing APP with Session ID
- Replace token with the token created
6) Run https with cert on different port http://example.com:8000
7) Visit the website with the html
8) Error on website
AnboxStream failed: failed to communicate with the signaler
9) Visit any session shows
It show: AnboxStream failed: failed to communicate with the signaler
10) Create new application and new session
It show: AnboxStream failed: failed to communicate with the signaler

Revision history for this message
Gary.Wang (gary-wzl77) wrote :

Thanks for those steps.

Could you run the following commands and share the output with us?

1. sudo snap get anbox-cloud-appliance experimental.location
2. juju ssh -m appliance:anbox-cloud anbox-stream-gateway/0 -- sudo cat /var/snap/anbox-stream-gateway/common/service/config.yaml | head -n5

BR
Gary

Revision history for this message
Terence (terencetwork) wrote :

I've followed the steps, and realised that both the location address is different, i've fixed the address and fix the https cert, currently the streaming it's working fine

Thanks Gary

Revision history for this message
Gary.Wang (gary-wzl77) wrote :

You're welcome. Good to hear you fixed that.
It's a bug in our documentation that if you don't want to reboot the machine after configuring the https cert and location for the appliance, you need to run the following command:

$ sudo systemctl start snap.anbox-cloud-appliance.reboot-checker.service

This enables the components like gateway/dashboard to use the location that you specified through `experimental.location` config in the end.

Gary.Wang (gary-wzl77)
Changed in anbox-cloud:
status: Triaged → Fix Committed
Revision history for this message
Terence (terencetwork) wrote :

Is there a nightly build for the patched version?

Does 1.15.0 still on track on August 24, 2022?

Revision history for this message
Simon Fels (morphis) wrote :

> Is there a nightly build for the patched version?

We currently only provide stable images, no nightly/beta/candidate.

> Does 1.15.0 still on track on August 24, 2022?

Yes, it is.

Revision history for this message
Terence (terencetwork) wrote :

ubuntu@ip-192-168-3-51:~$ anbox-cloud-appliance status
status: ready
update-available: false
reboot-needed: false

not able to update to 1.15.0

How can i check my current version?

Revision history for this message
Terence (terencetwork) wrote :

Somehow the update was possible, i've updated the code however,

"Anbox stream failed Error: connector failed to connect: failed to join session" When i launch camera app inside Safari IOS browser

The bug doesn't seems to be fixed

Revision history for this message
Gary.Wang (gary-wzl77) wrote :

Could you help to check if the application got updated after 1.15.0 images were imported to AMS? Something like,

$ amc application show <application_name>
$ amc image show $(amc application show <application_name> | grep "parent-image" | cut -d' ' -f2)

Could you share the output of the above commands with us?

Thanks.

BR
Gary

Revision history for this message
Terence (terencetwork) wrote :

It's working fine after i recreate the application
Thanks

Revision history for this message
Gary.Wang (gary-wzl77) wrote :

So looks like the application didn't get updated in time after the images were refreshed in AMS. It depends on the number of applications created in AMS and the free resources that your deployment owns to schedule a base container for an application update. Usually, after the parent images got updated(in your case, 1.14.x -> 1.15.0), AMS will schedule an update for each application and a new revision of the application will be generated automatically afterward.

Anyway, it's good to hear it's working fine now.

Changed in anbox-cloud:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.