novncproxy: code 404, message File not found

Poking at this a bit more this morning, I set debug=true in nova.conf, hoping to get more verbose output. I started novncproxy from the cli, and it's suddenly working.

6.0.1 is in HCF, high bugs can be assigned only for 6.0.2

Confirmed on 6.1 in bug: https://bugs.launchpad.net/fuel/+bug/1434562

Diagnostic snapshot:
https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaNjl3NE5VTTF6WTA/view?usp=sharing

I can see the same messages in logs:
http://paste.openstack.org/show/197175/

VNC itself works fine, I'm able to connect with vncviewer.

OK, it looks like our 'novnc' package is outdated. I got console working after downloading master branch from https://github.com/kanaka/noVNC to /usr/share/novnc/ on controller and restarting nova-novncproxy service. So we need to update 'novnc' package.
Forwarding this bug to OSCI team.

What do you mean by "outdated"? We have the latest 0.5.1 release for 6.1, which is known to be working:
https://bugs.launchpad.net/fuel/6.0.x/+bug/1418567

If you have some specific bugs that are fixed in master but are present in 0.5.1, please provide links to them - we will consider backporting.

OK. Here are some additional details. I can see 3 JS files are missing for novnc on Ubuntu envs:

# grep 404 /var/log/upstart/nova-novncproxy.log | grep 'GET /include/' | awk '{print $7}' | sort -u
/include/keyboard.js
/include/keysymdef.js
/include/keysym.js

# ll /usr/share/novnc/include/{keysymdef.js,keyboard.js,keysym.js}
ls: cannot access /usr/share/novnc/include/keysymdef.js: No such file or directory
ls: cannot access /usr/share/novnc/include/keyboard.js: No such file or directory
ls: cannot access /usr/share/novnc/include/keysym.js: No such file or directory

# dpkg -L novnc | grep /include/ | egrep 'keysymdef.js|keyboard.js|keysym.js' || echo Not found
Not found

Due to this, novnc in Horizon loads forever and I can't get console.

Checking our deb package:

# wget http://osci-mirror-msk.msk.mirantis.net/mos/ubuntu//pool/main/n/novnc/novnc_0.5.1+gitfc00821eba~mira6.1_all.deb
# dpkg -c novnc_0.5.1+gitfc00821eba~mira6.1_all.deb | grep /include/ | egrep 'keysymdef.js|keyboard.js|keysym.js' || echo Not found
Not found

CentOS rpm has those files in place so only Ubuntu is affected. As soon as I put those missing files into /usr/share/novnc/include/ on controllers, novnc console starts to work. So it looks like this problem is caused by missing files in out DEB package. Please check.

package novnc has been built for project packages/trusty/novnc
Package version == 1:0.5.1+gitfc00821eba~mira6.1, package release == ubuntu1

Changeset: https://review.fuel-infra.org/5098
project: packages/trusty/novnc
branch: 6.1
author: Aleksandr Mogylchenko
committer: Aleksandr Mogylchenko
subject: novncproxy: code 404, message File not found
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /trusty-fuel-6.1-stable-5098/

Could you please recheck newly built package?

Works fine on my local test env. Thanks.

Status of 6.0.x is not clear, so no fix this time

Verified on MOS 6.1 ISO #429
Steps to Verify:
root@node-1:~# ls /usr/share/novnc/include -1 | egrep 'keysymdef.js|keyboard.js|keysym.js'
keyboard.js
keysymdef.js
keysym.js

"Status of 6.0.x is not clear, so no fix this time" << does that mean we won't have fix for this issue in 6.0.x?

Fix proposed to branch: 6.0.1
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/9973

Reviewed: https://review.fuel-infra.org/9973
Submitter: Michael Semenov <email address hidden>
Branch: 6.0.1

Commit: 60b8fc2d4a277cfdbfea27d1ebe0b680839f40af
Author: Aleksandr Mogylchenko <email address hidden>
Date: Thu Jul 30 09:00:09 2015

novncproxy: code 404, message File not found

Some JS files were missing in the package, resulting in non-working novnc.

Change-Id: I8b67a598b8b1493876a837f851ec7b4bd1d8fa71
Closes-Bug: #1433894

6.0.1 status is not clear, but I've pushed the fix to our stable repos.

Confirmed on Fuel 10:

# tail /var/log/nova/nova-novncproxy.log
...
2017-04-07 15:19:09.622 153296 INFO nova.console.websocketproxy [-] 192.168.4.2 - - [07/Apr/2017 15:19:09] code 404, message File not found

Will mark this as "incomplete" while details (logs, cluster configs) are coming.

novnc in 10.0 shouldn't be affected. Marked as 'invalid'.

dpkg -c novnc_0.6.1-1~u16.04%2bmos1_all.deb | grep -E 'key(.*).js$'
-rw-r--r-- root/root 20840 2017-03-27 12:39 ./usr/share/novnc/include/keyboard.js
-rw-r--r-- root/root 22527 2017-03-27 12:39 ./usr/share/novnc/include/keysym.js
-rw-r--r-- root/root 21512 2017-03-27 12:39 ./usr/share/novnc/include/keysymdef.js