Contrail :: R4.0 16.04 131 ocata :: webui fails to come up.

The contrail-webui service does not come up due to node.js failing at startup. Seems to be a missing key file issue.

Setup:-

DISTRO : "Ubuntu 16.04.2 LTS"
SKU : ocata
Config Nodes : [u'nodem16', u'nodem17', u'nodem18']
Control Nodes : [u'nodem16', u'nodem17', u'nodem18']
Compute Nodes : [u'nodem19', u'nodem20', u'nodem5']
Openstack Node : [u'nodem16', u'nodem17', u'nodem18']
WebUI Node : [u'nodem16', u'nodem17', u'nodem18']
Analytics Node! s : [u'nodem16', u'nodem17', u'nodem18']
Database Nodes : [u'nodem16', u'nodem17', u'nodem18']
Physical Devices : [u'blr-mx2', u"'blr-mx2'"]
LB Nodes : [u'nodea10']

root@nodem16(controller):/# contrail-status
== Contrail Control ==
contrail-control: active
contrail-named: active
contrail-dns: active
contrail-control-nodemgr: active
== Contrail Config ==
contrail-api: active
contrail-schema: backup
contrail-svc-monitor: backup
contrail-device-manager: backup
contrail-config-nodemgr: active
== Contrail Config Database==
contrail-database: active

== Contrail Web UI ==
contrail-webui: inactive <<<<<<<< Contrail-WebUI service is inactive
contrail-webui-middleware: active
== Contrail Support Services ==
zookeeper: active
rabbitmq-server: inactive (disabled on boot)
root@nodem16(controller):/#

Internal Ansible Logs showing failure in bringing up WebUI service
**********************************************************************
**********************************************************************
**********************************************************************
**********************************************************************

Dec 17 07:49:11 nodem16 contrailctl[27]: TASK [contrail/webui : Make sure contrail-webui service is in desired state] ***
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: 12/17/2017 07:49:11 AM - debug: Built Package List as:[{"pkgName":null,"parseURL.xml":["/src/serverroot/common/url.routes.js","/src/serverroot/utils/url.routes.js","/src/serverroot/web/routes/url.routes.js","/webroot/reports/qe/api/url.routes.js","/webroot/reports/udd/api/url.routes.js"],"jobProcess.xml":["/src/serverroot/jobs/core/jobsCb.api.js"],"featureList.xml":["/src/serverroot/web/core/feature.list.js"]},{"pkgName":"webController","jobProcess.xml":["/webroot/monitor/infrastructure/controlnode/jobs/jobsCb.api.js","/webroot/monitor/infrastructure/vrouter/jobs/jobsCb.api.js","/webroot/monitor/networking/jobs/jobsCb.api.js"],"featureList.xml":["/webroot/common/api/feature.list.js"],"parseURL.xml":["/webroot/common/api/url.routes.js","/webroot/config/common/api/url.routes.js","/webroot/config/dns/records/api/url.routes.js","/webroot/config/dns/servers/api/url.routes.js","/webroot/config/infra/bgp/api/url.routes.js","/webroot/config/infra/linklocalservices/api/url.routes.js","/webroot/config/infra/quotas/api/url.routes.js","/webroot/config/infra/serviceappliance/api/url.routes.js","/webroot/config/infra/serviceapplianceset/api/url.routes.js","/webroot/config/infra/vrouters/api/url.routes.js","/webroot/config/networking/fip/api/url.routes.js","/webroot/config/networking/ipam/api/url.routes.js","/webroot/config/networking/logicalrouter/api/url.routes.js","/webroot/config/networking/networks/api/url.routes.js","/webroot/config/networking/policy/api/url.routes.js","/webroot/config/networking/port/api/url.routes.js","/webroot/config/networking/routetable/api/url.routes.js","/webroot/config/networking/routingpolicy/api/url.routes.js","/webroot/config/networking/securitygroup/api/url.routes.js","/webroot/config/physicaldevices/interfaces/api/url.routes.js","/webroot/config/physicaldevices/physicalrouters/api/url.routes.js","/webroot/config/port/api/url.routes.js","/webroot/config/services/bgpasaservice/api/url.routes.js","/webroot/config/services/instances/api/url.routes.js","/webroot/config/services/templates
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: /api/url.routes.js","/webroot/monitor/infrastructure/analyticsnode/api/url.routes.js","/webroot/monitor/infrastructure/confignode/api/url.routes.js","/webroot/monitor/infrastructure/controlnode/api/url.routes.js","/webroot/monitor/infrastructure/databasenode/api/url.routes.js","/webroot/monitor/infrastructure/globalcontroller/api/url.routes.js","/webroot/monitor/infrastructure/underlay/api/url.routes.js","/webroot/monitor/infrastructure/vrouter/api/url.routes.js","/webroot/monitor/networking/api/url.routes.js","/webroot/setting/configdb/ui/api/url.routes.js"]}]
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: fs.js:439
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: ^
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: Error: ENOENT, no such file or directory './keys/cs-key.pem'
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: at Object.fs.openSync (fs.js:439:18)
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: at Object.fs.readFileSync (fs.js:290:15)
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: at Object.<anonymous> (/usr/src/contrail/contrail-web-core/webServerStart.js:96:12)
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: at Module._compile (module.js:456:26)
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: at Object.Module._extensions..js (module.js:474:10)
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: at Module.load (module.js:356:32)
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: at Function.Module._load (module.js:312:12)
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: at Function.Module.runMain (module.js:497:10)
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: at startup (node.js:119:16)
Dec 17 07:49:11 nodem16 contrailWebServer.sh[7428]: at node.js:929:3
Dec 17 07:49:11 nodem16 systemd[1]: contrail-webui.service: Main process exited, code=exited, status=8/n/a
Dec 17 07:49:11 nodem16 systemd[1]: contrail-webui.service: Unit entered failed state.
Dec 17 07:49:11 nodem16 systemd[1]: contrail-webui.service: Failed with result 'exit-code'.
Dec 17 07:49:11 nodem16 systemd[1]: contrail-webui.service: Service hold-off time over, scheduling restart.
Dec 17 07:49:11 nodem16 systemd[1]: Stopped "Contrail Web UI".
Dec 17 07:49:11 nodem16 systemd[1]: Started "Contrail Web UI".
Dec 17 07:49:11 nodem16 systemd[1]: Configuration file /lib/systemd/system/contrail-webui.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Dec 17 07:49:11 nodem16 systemd[1]: Configuration file /lib/systemd/system/contrail-webui.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Dec 17 07:49:11 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:11 AM - debug: Redis: connect
Dec 17 07:49:11 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:11 AM - debug: Redis: ready
Dec 17 07:49:11 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:11 AM - debug: Redis FLUSHDB 2 done
Dec 17 07:49:11 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:11 AM - debug: Redis FLUSHDB 3 done
Dec 17 07:49:11 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:11 AM - debug: Redis FLUSHDB 4 done
Dec 17 07:49:11 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:11 AM - debug: Redis FLUSHDB 5 done
Dec 17 07:49:11 nodem16 systemd[1]: Configuration file /lib/systemd/system/contrail-webui.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Dec 17 07:49:11 nodem16 systemd[1]: Reloading.
Dec 17 07:49:11 nodem16 systemd[1]: Configuration file /lib/systemd/system/contrail-database.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Dec 17 07:49:11 nodem16 systemd[1]: [/lib/systemd/system/contrail-database.service:10] Not an absolute path, ignoring: ${CASSANDRA_USER_HOME}/cassandra.pid
Dec 17 07:49:11 nodem16 systemd[1]: Configuration file /lib/systemd/system/contrail-webui.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Dec 17 07:49:11 nodem16 systemd[1]: Configuration file /lib/systemd/system/contrail-webui-middleware.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Dec 17 07:49:12 nodem16 contrailctl[27]: changed: [localhost]
Dec 17 07:49:12 nodem16 contrailctl[27]: TASK [contrail/webui : Make sure contrail-webui-middleware service is in desired state] ***
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Built Package List as:[{"pkgName":null,"parseURL.xml":["/src/serverroot/common/url.routes.js","/src/serverroot/utils/url.routes.js","/src/serverroot/web/routes/url.routes.js","/webroot/reports/qe/api/url.routes.js","/webroot/reports/udd/api/url.routes.js"],"jobProcess.xml":["/src/serverroot/jobs/core/jobsCb.api.js"],"featureList.xml":["/src/serverroot/web/core/feature.list.js"]},{"pkgName":"webController","jobProcess.xml":["/webroot/monitor/infrastructure/controlnode/jobs/jobsCb.api.js","/webroot/monitor/infrastructure/vrouter/jobs/jobsCb.api.js","/webroot/monitor/networking/jobs/jobsCb.api.js"],"featureList.xml":["/webroot/common/api/feature.list.js"],"parseURL.xml":["/webroot/common/api/url.routes.js","/webroot/config/common/api/url.routes.js","/webroot/config/dns/records/api/url.routes.js","/webroot/config/dns/servers/api/url.routes.js","/webroot/config/infra/bgp/api/url.routes.js","/webroot/config/infra/linklocalservices/api/url.routes.js","/webroot/config/infra/quotas/api/url.routes.js","/webroot/config/infra/serviceappliance/api/url.routes.js","/webroot/config/infra/serviceapplianceset/api/url.routes.js","/webroot/config/infra/vrouters/api/url.routes.js","/webroot/config/networking/fip/api/url.routes.js","/webroot/config/networking/ipam/api/url.routes.js","/webroot/config/networking/logicalrouter/api/url.routes.js","/webroot/config/networking/networks/api/url.routes.js","/webroot/config/networking/policy/api/url.routes.js","/webroot/config/networking/port/api/url.routes.js","/webroot/config/networking/routetable/api/url.routes.js","/webroot/config/networking/routingpolicy/api/url.routes.js","/webroot/config/networking/securitygroup/api/url.routes.js","/webroot/config/physicaldevices/interfaces/api/url.routes.js","/webroot/config/physicaldevices/physicalrouters/api/url.routes.js","/webroot/config/port/api/url.routes.js","/webroot/config/services/bgpasaservice/api/url.routes.js","/webroot/config/services/instances/api/url.routes.js","/webroot/config/services/templates
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: /api/url.routes.js","/webroot/monitor/infrastructure/analyticsnode/api/url.routes.js","/webroot/monitor/infrastructure/confignode/api/url.routes.js","/webroot/monitor/infrastructure/controlnode/api/url.routes.js","/webroot/monitor/infrastructure/databasenode/api/url.routes.js","/webroot/monitor/infrastructure/globalcontroller/api/url.routes.js","/webroot/monitor/infrastructure/underlay/api/url.routes.js","/webroot/monitor/infrastructure/vrouter/api/url.routes.js","/webroot/monitor/networking/api/url.routes.js","/webroot/setting/configdb/ui/api/url.routes.js"]}]
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis: connect
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis: connect
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis: connect
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis: connect
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis: ready
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis: ready
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis: ready
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis: ready
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis FLUSHDB 2 done
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis FLUSHDB 3 done
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis FLUSHDB 4 done
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: Redis FLUSHDB 5 done
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: /usr/lib64/node_modules/redis/index.js:541
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: throw err;
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: ^
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: Error: ENOENT, no such file or directory './keys/cs-key.pem'
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: at Object.fs.openSync (fs.js:439:18)
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: at Object.fs.readFileSync (fs.js:290:15)
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: at Object.<anonymous> (/usr/lib64/node_modules/kue/lib/http/index.js:44:12)
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: at Module._compile (module.js:456:26)
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: at Object.Module._extensions..js (module.js:474:10)
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: at Module.load (module.js:356:32)
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: at Function.Module._load (module.js:312:12)
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: at Module.require (module.js:364:17)
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: at require (module.js:380:17)
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: at Function.Object.defineProperty.get (/usr/lib64/node_modules/kue/lib/kue.js:48:26)
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - debug: The worker #1 has disconnected.
Dec 17 07:49:12 nodem16 contrailWebMiddleware.sh[7458]: 12/17/2017 07:49:12 AM - error: The worker #1 has exited with exit code 8