OpenStack Compute (nova)

LibvirtDriver: "get_serial_console" returns always the first port of N defined ports

Bug #1425640 reported by Markus Zoeller (markus_z)
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Medium
Unassigned

Bug Description

Issue
=====
The libvirt driver returns *always" the first port and ignores each
further defined port for the "serial console" feature.

Steps to reproduce
==================

* enable the "serial console" feature in nova.conf
  [serial_console]
  enabled=True
* start the nova-serialproxy service (e.g. with devstack [2])
* append the extra_spec "hw:serial_port_count" with value "2" to a flavor
* start an instancen with that flavor (for generated domain XML see [4])
* Connect to the serial console to that launched instance
  (e.g. Horizon with "console" tab or another client [3])
* Start a second attempt to connect to that instance

Expected behavior
=================

The first connection will use the default port 10000.
The second attempt will use the next available port 10001.

Actual behavior
===============

The second attemp will fail because the libvirt driver returns the
already connected port 10000.

Logs & Env.
===========

The two attempts to connect happen at 19:04 and 19:09 in the logs.
Please see [6] and [7] for the relevant nova-serialproxy and nova-compute
logs. The additional dumpxml [5] from libvirt is taken after the first
connection.
I'm using libvirt version 1.2.2 within an Ubuntu14.04 VM. OpenStack
is installed and started with devstack.

References
==========

[1] Mailing list "openstack-dev"; February 2015;
    subject "[nova] bp serial-ports *partly* implemented?";
    http://markmail.org/thread/2byi6adf4ynvjg6e
[2] Devstack guide; Nova and devstack;
    http://docs.openstack.org/developer/devstack/guides/nova.html
[3] larsk/novaconsole; github; https://github.com/larsks/novaconsole/
[4] pastebin; generated domain XML for the instance;
    http://paste.openstack.org/show/182024/
[5] pastebin; virsh dumpxml <domain> for the instance after the first
    connection; http://paste.openstack.org/show/182026/
[6] pastebin; log of the nova-serialproxy service
    http://paste.openstack.org/show/182028/
[7] pastebin; log of nova-compute service
    http://paste.openstack.org/show/182029/

Sahid Orentino (sahid-ferdjaoui)
Changed in nova:
assignee: nobody → sahid (sahid-ferdjaoui)
Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

The current code try to find an update of the attribute 'mode'(from bind to connect) to see if it indicate that the console is connected, but according to the xml you are providing when a client is connected. libvirt do not change the state of mode.

Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

What configuration of libvirt is expected to be able to do that switch from "bind" to "connect"?

Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

libvirt does not inform that, the console is connected. We need to track this information between process (api, compute, serialproxy) so we cannot simply use an in-memory solution.

The simplest way could be to have libvirt handle this information or we need to track it with a database or something shared between process.

Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

After thinking little bit more, a way could be to update the actual API to let it to pass an arguement port-number and add a new API to return a list of serial-port created.

An other way would be to have a way to set a new attribute to the connection info which can be used to understand if yes or no the port for this connection is used and then store it to the database. The current code store the connection_info in memory or in memcached if available.

Looking at the code, we need to update Console classes to use the objects framework and let them handle the connection_info also remove the way that we use to store it in memory to use a real database. This is more than a but and a spec needs to spec provided for lemming.

I will work on it.

Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

@sahid (sahid-ferdjaoui):
This bug is in progress for more than 30 days without a patch. To give other contributors a chance to work on that I set it as "unassigned". Feel free to reassign yourself if you like to continue to work on this bug. If you do so, please set the status to "in progress".

Changed in nova:
assignee: sahid (sahid-ferdjaoui) → nobody
Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

Still working on it but some work have to be done before we can address it

https://review.openstack.org/#/c/165838/

Changed in nova:
assignee: nobody → sahid (sahid-ferdjaoui)
Revision history for this message
Nikola Đipanov (ndipanov) wrote :

So it seems to me the real bug is in libvirt driver get_serial_console() method.

https://github.com/openstack/nova/blob/f729a1e815f9b7fcfc91fb2b3c2cabbe3f4af7d5/nova/virt/libvirt/driver.py#L2585-L2587

That for loop there will obviously always run only once. That code smells from a mile away.

We should really be returning all posible console connections, authorize them all and let the user choose how to connect to them.

Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

The problem is that libvirt does not flag to "bind" when a console is connected so we need to have something persistent and able to be shared between services.

Markus Zoeller (markus_z) (mzoeller)
tags: added: console
Davanum Srinivas (DIMS) (dims-v)
Changed in nova:
assignee: sahid (sahid-ferdjaoui) → nobody
stgleb (gstepanov)
Changed in nova:
assignee: nobody → stgleb (gstepanov)
status: Confirmed → In Progress
Revision history for this message
Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

Changed in nova:
status: In Progress → Confirmed
assignee: stgleb (gstepanov) → nobody
Revision history for this message
Matt Riedemann (mriedem) wrote :

With blueprint https://blueprints.launchpad.net/nova/+spec/convert-consoles-to-objects complete in Rocky we now have persistent console information and should be able to resolve this bug.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.