skyline web no domian select

1. please supply the info about command: docker inspect skyline|grep "skyline."
2. please supply the info about command: openstack domain list
3. please supply the log file about skyline

thanks

OK ,I modified the configuration file again and rebooted the configuration.

User name, database has been created

#conf

root@controller:~# cat skyline.yaml
default:
  access_token_expire: 3600
  access_token_renew: 1800
  cors_allow_origins: []
  database_url: mysql+pymysql://skyline:skyline@controller/skyline
  debug: false
  log_dir: ./log
  log_file: skyline.log
  prometheus_basic_auth_password: ''
  prometheus_basic_auth_user: ''
  prometheus_enable_basic_auth: false
  prometheus_endpoint: http://localhost:9091
  secret_key: aCtmgbcUqYUy_HNVg5BDXCaeJgJQzHJXwqbXr0Nmb2o
  session_name: session
  ssl_enabled: true
openstack:
  base_domains:
  - heat_user_domain
  default_region: RegionOne
  enforce_new_defaults: true
  extension_mapping:
    floating-ip-port-forwarding: neutron_port_forwarding
    fwaas_v2: neutron_firewall
    qos: neutron_qos
    vpnaas: neutron_vpn
  interface_type: public
  keystone_url: http://192.168.100.10:5000/v3/
  nginx_prefix: /api/openstack
  reclaim_instance_interval: 604800
  service_mapping:
    baremetal: ironic
    compute: nova
    container: zun
    container-infra: magnum
    database: trove
    identity: keystone
    image: glance
    key-manager: barbican
    load-balancer: octavia
    network: neutron
    object-store: swift
    orchestration: heat
    placement: placement
    sharev2: manilav2
    volumev3: cinder
  sso_enabled: false
  sso_protocols:
  - openid
  sso_region: RegionOne
  system_admin_roles:
  - admin
  - system_admin
  system_project: service
  system_project_domain: Default
  system_reader_roles:
  - system_reader
  system_user_domain: Default
  system_user_name: skyline
  system_user_password: 'skyline'
setting:
  base_settings:
  - flavor_families
  - gpu_models
  - usb_models
  flavor_families:
  - architecture: x86_architecture
    categories:
    - name: general_purpose
      properties: []
    - name: compute_optimized
      properties: []
    - name: memory_optimized
      properties: []
    - name: high_clock_speed
      properties: []
  - architecture: heterogeneous_computing
    categories:
    - name: compute_optimized_type_with_gpu
      properties: []
    - name: visualization_compute_optimized_type_with_gpu
      properties: []
  gpu_models:
  - nvidia_t4
  usb_models:
  - usb_c

log(s)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/gunicorn", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/app/wsgiapp.py", line 67, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/app/base.py", line 231, in run
    super().run()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/arbiter.py", line 229, in run
    self.halt(reason=inst.reason, exit_status=inst.exit_status)
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/arbiter.py", line 342, in halt
    self.stop()
  File "/usr/local/lib/python3.8/dist-packages/gunicorn/arbiter.py", line 393, in stop
    time.sl...

i have the same problem i am able to open skyline dashboard but when i want to login the select domain & region section is empty it returns no data :-(
the database has 3 tables that's it.

yes! databases is null !

I doubt you had not completed the preparations before you started the service.
FYI: https://docs.openstack.org/skyline-apiserver/latest/install/docker-install-ubuntu.html

Please verify you have been created skyline database, skyline user and grant permissions corectly, And verify you have been create keystone user, service and endpoint of skyline before you start the skyline.

Insides, you should add port in the configuration 'database_url' in skyline.yaml.

https://bugs.launchpad.net/skyline-apiserver/+bug/1992323

Did you meet the same scene ? Set the keystone service name not as "keystone" ?

I have the same issue - installed from kolla-ansible - DB looks fine, all pre-reqs.. i get this in the error log.

[2023-06-02 21:35:25 -0400].355 734 WARNING [-] Invalid HTTP request received.
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 129, in handle_events
    event = self.conn.next_event()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_connection.py", line 443, in next_event
    exc._reraise_as_remote_protocol_error()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
    raise self
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_connection.py", line 425, in next_event
    event = self._extract_next_receive_event()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
    event = self._reader(self._receive_buffer)
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_readers.py", line 68, in maybe_read_from_IDLE_client
    raise LocalProtocolError("illegal request line")
h11._util.RemoteProtocolError: illegal request line

No domain etc shown in the GUI.

Version 2023.1 btw - current code. IS there some bug or issue i should be aware of and am missing ?

Hi, Noel

Have you enabled tls for any endpoints?
Or you could show me your results of command `openstack endpoint list`.

Had the same problem seems it's related to enabled TLS ( kolla-ansible, self-singed certs, probably ca.crt is not imported correctly to containers cacert, there wasn't any mention of insecure connection in Skyline logs ) for endpoints.
It's working ok with certs signed from Lets Encrypt.

YEs - I use self signed TLS.... is it not being checked correct in the docker?

There are endpoints (I hid the URL for security reasons)

(kolla-2023.1) root@cube-server:~/kolla-2023.1/TN_DEV_NY_5_NET# openstack endpoint list | grep skyline | awk '{print $4,$6,$12}'
TN_DEV_NY_5_NET skyline internal
TN_DEV_NY_5_NET skyline public
TN_DEV_NJ_55_NET skyline public
TN_DEV_NJ_55_NET skyline internal

Pretty sure the issue is TLS. Will verify & check shortly

Well, I can hit keystone fine and there's a root cert there... it was copied, seems the skyline code must be failing somehow TLS wise.... any ideas ? IT MUST be failing to talk to keystone but i can hit keystone fine w a curl, no -k or anything on https

(skyline-apiserver)[root@tunninet-server-noel /]$ curl https://int.noel.openstack.tunninet.com:5000/v3
{"version": {"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "https://int.noel.openstack.tunninet.com:5000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}}curl (https://int.noel.openstack.tunninet.com:5000/v3): response: 200, time: 0.064707, size: 272

(skyline-console)[root@tunninet-server-noel /]$ curl https://int.noel.openstack.tunninet.com:5000/v3
{"version": {"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "https://int.noel.openstack.tunninet.com:5000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}}curl (https://int.noel.openstack.tunninet.com:5000/v3): response: 200, time: 0.062545, size: 272

(skyline-console)[root@tunninet-server-noel /]$ ls -la /usr/local/share/ca-certificates
total 16
drwxr-xr-x 1 root root 4096 Jul 4 22:19 .
drwxr-xr-x 1 root root 4096 Jul 3 23:33 ..
-rw-r--r-- 1 root root 1814 Jul 4 22:19 kolla-customca-root.crt
(skyline-console)[root@tunninet-server-noel /]$ ls -la /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 210381 Jul 4 22:19 /etc/ssl/certs/ca-certificates.crt

(skyline-apiserver)[root@tunninet-server-noel /]$ ls -la /usr/local/share/ca-certificates
total 16
drwxr-xr-x 1 root root 4096 Jul 4 22:19 .
drwxr-xr-x 1 root root 4096 Jul 3 23:28 ..
-rw-r--r-- 1 root root 1814 Jul 4 22:19 kolla-customca-root.crt
(skyline-apiserver)[root@tunninet-server-noel /]$ ls -la /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 210381 Jul 4 22:19 /etc/ssl/certs/ca-certificates.crt

def the issue is in the skyline code not using the system certs for TLS somehow... (my best guess) seems like it can not get domain info from keystone w a self signed....

Thanks for your verification, and I'd like to figure out the following two questions:
1. which kind of endpoints of skyline were enabled tls? just public? or internal or admin? or all of them?
2. when the error was raised? in the process of service starting? or in the calling when you entered the login website?

then maybe we are supposed to verify that, doubting there are some bugs

Kolla Ansible deploys TLS on every endpoint by default (public and private). Every service must talk to every other service using TLS, if you self sign the cert, it will deploy your root CA and 99% of the time where things hit a snag is that strict checking is on and the service in question does not use the custom root CA to verify the cert.

Here, I believe the issue is that skyline is trying to talk to keystone to get a token, domains etc and it fails there. Logging wise, this is all i see (The log indicates a protocol error - assumed = https)

raise LocalProtocolError("illegal request line")
h11._util.RemoteProtocolError: illegal request line <---

==> /var/log/kolla/skyline/skyline-access.log <==

==> /var/log/kolla/skyline/skyline-error.log <==
    exc._reraise_as_remote_protocol_error()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
    raise self
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_connection.py", line 425, in next_event
    event = self._extract_next_receive_event()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
    event = self._reader(self._receive_buffer)
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_readers.py", line 68, in maybe_read_from_IDLE_client
    raise LocalProtocolError("illegal request line")
h11._util.RemoteProtocolError: illegal request line

==> /var/log/kolla/skyline/skyline.log <==
2023-07-04 22:19:14.214 | INFO | databases.core:connect:90 - Connected to database mysql://skyline:********@int.noel.openstack.tunninet.com:3306/skyline
2023-07-04 22:19:14.256 | INFO | databases.core:connect:90 - Connected to database mysql://skyline:********@int.noel.openstack.tunninet.com:3306/skyline
2023-07-04 22:19:14.360 | INFO | databases.core:connect:90 - Connected to database mysql://skyline:********@int.noel.openstack.tunninet.com:3306/skyline
2023-07-04 22:19:14.360 | INFO | databases.core:connect:90 - Connected to database mysql://skyline:********@int.noel.openstack.tunninet.com:3306/skyline
2023-07-04 22:19:14.375 | INFO | databases.core:connect:90 - Connected to database mysql://skyline:********@int.noel.openstack.tunninet.com:3306/skyline

==> /var/log/kolla/skyline/skyline-nginx-access.log <==
192.168.5.1 - - [06/Jul/2023:08:41:14 -0400] "0.000" "-" "GET /docs HTTP/2.0" 200 424 "-" "curl-healthcheck" "-"
192.168.5.1 - - [06/Jul/2023:08:41:44 -0400] "0.000" "-" "GET /docs HTTP/2.0" 200 424 "-" "curl-healthcheck" "-"
192.168.5.1 - - [06/Jul/2023:08:42:14 -0400] "0.000" "-" "GET /docs HTTP/2.0" 200 424 "-" "curl-healthcheck" "-"
192.168.5.1 - - [06/Jul/2023:08:42:44 -0400] "0.000" "-" "GET /docs HTTP/2.0" 200 424 "-" "curl-healthcheck" "-"
192.168.5.1 - - [06/Jul/2023:08:43:14 -0400] "0.000" "-" "GET /docs HTTP/2.0" 200 424 "-" "curl-healthcheck" "-"
192.168.5.1 - - [06/Jul/2023:08:43:44 -0400] "0.000" "-" "GET /docs HTTP/2.0" 200 424 "-" "curl-healthcheck" "-"
192.168.5.1 - - [06/Jul/2023:08:44:14 -0400] "0.000" "-" "GET /docs HTTP/2.0" 200 424 "-" "curl-healthcheck" "-"
192.168.5.1 - - [06/Jul/2023:08:44:45 -0400] "0.0...

@Shuai - Is there a way we can disable strict cert check or (better yet) tell it to try the cert in the os to validate it ? ie /etc/ssl/certs/ca-bundle.crt ? Seemingly this is where the problem is appearing.

we try to reproduce the issue, deploying in kolla-ansible, enable the external and internal tls, but we do get the domain list.

maybe the office doc about tls in kolla-ansible could give you help.
https://docs.openstack.org/kolla-ansible/latest/admin/tls.html

Have ben running open stack with self signed without issue for a long time, all options are set from that doc, but still, no dropdown. The issue seems to me like we need a way to disable strict check or that it is not using my CA cert somewhere.

(kolla-2023.1) root@cube-server:~# grep -i tls globals.yml | grep -v \# | grep kolla
kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "{{ kolla_enable_tls_internal if kolla_same_external_internal_vip | bool else 'no' }}"
kolla_enable_tls_backend: "yes"
kolla_verify_tls_backend: "yes"
kolla_tls_backend_cert: "{{ kolla_certificates_dir }}/backend-cert.pem"
kolla_tls_backend_key: "{{ kolla_certificates_dir }}/backend-key.pem"

(kolla-2023.1) root@cube-server:~# grep -i copy globals.yml | grep -v \# | grep kolla
kolla_copy_ca_into_containers: "yes"

I also did check the container itself and the cert is there and copied...

I'm really sorry that couldn't get some solution in the situation based on info above, maybe re-deploy skyline would be helpful.
Or you could debug and find the poistion the error raised in skyline in your environment.
We would get improved about logging in the future.

Hello, I have deployed it on 5 different servers - a consistent issue. Is there any way i can via cli test where it may fail on a self signed cert ?

Same issue. I just pulled skyline kolla images. I'm using self-signed certs. I also see the skyline_apiserver image in the unhealthy state with the logs file containing the abovementioned errors. Could someone confirm if it is working fine with lets encrypt?

Have you guys found a way to resolve this issue? I built an image using Kolla-build with the latest version (master branch) of the Skyline-Apiserver and deployed it. I see that the issue still persists! I'm now using lets-encrypt for external VIP encryption and self-signed for internal services and private VIP. Horizon works perfectly without any issue!

hello Debasis, sorry for response lately because of developping task recently.

I have deployed a master and a 2023.1 release with kolla-ansible,
and finally re-produce the issue in the 2023.1 release and get the same error.
master is fine, and you had tested skyline in master branch, so I think that there might be other service cause this error but skyline itself.

we'll debug and give you response ASAP.

Thank you for your response! I think, it's an issue with skyline apiserver(state is unhealthy). It works fine when used without tls encription. The logs gives me the following error for your reference. Anything else needed, let me know....

h11._util.RemoteProtocolError: illegal request line
[2023-08-23 04:25:13 +0530].235 735 WARNING [-] Invalid HTTP request received.
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 129, in handle_events
    event = self.conn.next_event()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_connection.py", line 487, in next_event
    exc._reraise_as_remote_protocol_error()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_util.py", line 77, in _reraise_as_remote_protocol_error
    raise self
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_connection.py", line 469, in next_event
    event = self._extract_next_receive_event()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_connection.py", line 411, in _extract_next_receive_event
    event = self._reader(self._receive_buffer)
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_readers.py", line 79, in maybe_read_from_IDLE_client
    raise LocalProtocolError("illegal request line")
h11._util.RemoteProtocolError: illegal request line
[2023-08-23 04:25:13 +0530].235 737 WARNING [-] Invalid HTTP request received.
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 129, in handle_events
    event = self.conn.next_event()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_connection.py", line 487, in next_event
    exc._reraise_as_remote_protocol_error()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_util.py", line 77, in _reraise_as_remote_protocol_error
    raise self
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_connection.py", line 469, in next_event
    event = self._extract_next_receive_event()
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_connection.py", line 411, in _extract_next_receive_event
    event = self._reader(self._receive_buffer)
  File "/var/lib/kolla/venv/lib/python3.10/site-packages/h11/_readers.py", line 79, in maybe_read_from_IDLE_client
    raise LocalProtocolError("illegal request line")
h11._util.RemoteProtocolError: illegal request line

we found the reason causes the issue.

maybe kolla-ansible changed the way for verifing ssl in the haproxy in 2023.1, it turns out that it didn't make it perfectly. Then we'll post issue for this in kolla-ansible.

And now the tmp resolution for you is that
1. add the following 2 lines into /etc/kolla/skyline-apiserver/gunicorn.py
(these two files should be in skyline_apiserver container, generated by kolla-ansible.)
```
keyfile = "/etc/skyline/certs/skyline-key.pem"
certfile = "/etc/skyline/certs/skyline-cert.pem"
```
then /etc/kolla/skyline-apiserver/gunicorn.py is like
# something
bind = "172.16.150.185:9998"
workers = 5
worker_class = "uvicorn.workers.UvicornWorker"
timeout = 300
keepalive = 5
reuse_port = True
proc_name = "skyline"
keyfile = "/etc/skyline/certs/skyline-key.pem"
certfile = "/etc/skyline/certs/skyline-cert.pem"
# something

2. restart skyline-apiserver container.

Thank you! You saved me!