kolla issue with multiple nova_consoleauth and HA
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| kolla |
Fix Released
|
Critical
|
Sam Yaple | ||
Bug Description
I believe I have run into this issue here:
https:/
In a test setup with 2 x control nodes and 2 x compute nodes, attempts to access the console of an instance only succeed 50% of the time.
I found if I shut down one of the nova_consoleauth containers, then it worked 100% of the time.
One workaround seems to be to adjust the inventory to do just that, by including a specific host here, vs 'nova' which it is typically. (I have not yet actually tested this workaround) - but definitely, manually stopping one of the nova_consoleauth containers prevents the failures.
[nova-consoleau
control01
Another potential way to address this is to bring memcache into the mix. By having memcache in the configuration here should also circumvent the issue.
| Changed in kolla: | |
| status: | New → Triaged |
| importance: | Undecided → High |
| Sam Yaple (s8m) wrote | #1 |
| Changed in kolla: | |
| importance: | High → Critical |
| assignee: | nobody → Sam Yaple (s8m) |
| milestone: | none → liberty-rc2 |
| Changed in kolla: | |
| milestone: | liberty-rc2 → liberty-rc3 |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to kolla (master) | #2 |
| Changed in kolla: | |
| status: | Triaged → In Progress |
| Changed in kolla: | |
| milestone: | liberty-rc3 → mitaka-1 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to kolla (master) | #3 |
| Changed in kolla: | |
| status: | In Progress → Fix Committed |
| OpenStack Infra (hudson-openstack) wrote Fix proposed to kolla (stable/liberty) | #4 |
| OpenStack Infra (hudson-openstack) wrote Fix merged to kolla (stable/liberty) | #5 |
| tags: | added: in-stable-liberty |
| Changed in kolla: | |
| status: | Fix Committed → Fix Released |
Sorry guys. been meaning to fix this. I brought this up in the HA documents. There are two ways to fix this.
1. Setup HAproxy as source balanced for Horizon and nova-consoleauth
2. Setup Horizon and nova-consoleauth to use memcached
I do not like memcached due to its nature. It has no security and with a single oneliner you can rip all of the valid tokens out of a memcached server. Since most environments are not configured correctly from a network security point of view what ends up happening is that from an unprivileged guest in a VM I can get undetected admin access to the entire OpenStack environment.
As such, I am strongly against making anything memcache the default option, but I am ok with making it configurable. I will work up the source patch today and we can discuss it on Wednesday.