OSA doesn't handle hosts file in compute nodes for live migration

Bug #1576724 reported by Jirayut Nimsaeng
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
openstack-ansible
Fix Committed
Low
Andrew Bonney

Bug Description

For live migration. All compute nodes need to have hosts file for each others. OSA still not handle this and I have to manually configure it by hand.

Revision history for this message
Bjoern (bjoern-t) wrote :

OSA does manage the hosts inside the openstack_hosts role via the openstack-host-hostfile tag.
Perhaps you just run openstack-hosts-setup.yml with the --tags openstack-host-hostfile again.
This issue usually happens when --limit is used to run playbooks.

Matt Thompson (mattt416)
Changed in openstack-ansible:
assignee: nobody → Matt Thompson (mattt416)
Revision history for this message
Jirayut Nimsaeng (winggundamth) wrote :

@Bjoern Actually it does update hosts file but with the name that you used in openstack_user_config.yml file. So you need to use compute node to be the same name as your hostname. Specially on compute node it does use the management ip but it does use the ssh ip instead. I'm not sure this is the proper way or not?

Action for this bug should be
- Properly document about how to update hosts file like after you adding new compute node or generally how to update hosts file
- Properly document that you have to configure openstack_user_config.yml on compute node that use the same name as you configure hostname on all compute nodes
- If live migration should use management ip instead of ssh ip so we have to adding this to the role openstack_hosts

Revision history for this message
Matt Thompson (mattt416) wrote :

Hi Jirayut,

I agree that we should probably make it clear that your inventory should reflect the actual hostnames assigned to the nodes. I'm a bit unclear about your management net IP / ssh IP though. Can you please elaborate? From my understanding, the management net IP is used for ssh'ing, so unless you've configured that differently the two should be the same.

Thanks!

--Matt

Revision history for this message
Kevin Carter (kevin-carter) wrote :

@Matt / @Jirayut,

Whatever the solution we come up to resolve this in a programmatic fashion we need to be aware that long hostnames will cause problems with SSH. We currently have a guard in inventory to ensure we don't hit those problems [0] but in solving for "live migration" we may need to be creative in how this is resolved.

[0] - https://github.com/openstack/openstack-ansible/blob/master/playbooks/inventory/dynamic_inventory.py#L333-L339

Revision history for this message
Jirayut Nimsaeng (winggundamth) wrote :

@Matt, sorry for delay. Refer to this architecture http://docs.openstack.org/developer/openstack-ansible/mitaka/install-guide/targethosts-networkexample.html

The SSH IP is bond0 ip address 10.240.0.11
The Management IP is br-mgmt ip address 172.29.236.11

@Kevin thank you for the problem that you aware of.

Changed in openstack-ansible:
status: New → Confirmed
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-os_nova (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/741155

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_nova (master)

Reviewed: https://review.opendev.org/741155
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=71f3fb224a1c4cac0490182543fe20bb8b64d690
Submitter: Zuul
Branch: master

commit 71f3fb224a1c4cac0490182543fe20bb8b64d690
Author: Andrew Bonney <email address hidden>
Date: Wed Jul 15 10:48:37 2020 +0100

    Use Nova management IP for live migrations

    Our installation uses separate networks for deployment and for
    OpenStack management. By default, live migrations depend upon a
    hostname, which resolves to an IP on our (lower speed) deployment
    network. Dependent on the deployment, the hostname may not be a
    reliable mechanism for determining the correct network for live
    migrations.

    Nova exposes a live_migration_inbound_addr which can be used to
    correct this. This patch defaults the inbound_addr to match the
    Nova service bind address, but provides the option for it to be
    overwritten through variables.

    Whilst the Nova docs suggest that live_migration_inbound_addr
    is ignored when live_migration_tunnelled is enabled this appears
    to be inaccurate from our testing.

    Change-Id: Iff6326f72971364d275ea999418d476007690ef8
    Related-Bug: #1576724

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-os_nova (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/744396

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_nova (stable/ussuri)

Reviewed: https://review.opendev.org/744396
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=2cffce8485ea9db465ed7bbc2121e55d76d017ca
Submitter: Zuul
Branch: stable/ussuri

commit 2cffce8485ea9db465ed7bbc2121e55d76d017ca
Author: Andrew Bonney <email address hidden>
Date: Wed Jul 15 10:48:37 2020 +0100

    Use Nova management IP for live migrations

    Our installation uses separate networks for deployment and for
    OpenStack management. By default, live migrations depend upon a
    hostname, which resolves to an IP on our (lower speed) deployment
    network. Dependent on the deployment, the hostname may not be a
    reliable mechanism for determining the correct network for live
    migrations.

    Nova exposes a live_migration_inbound_addr which can be used to
    correct this. This patch defaults the inbound_addr to match the
    Nova service bind address, but provides the option for it to be
    overwritten through variables.

    Whilst the Nova docs suggest that live_migration_inbound_addr
    is ignored when live_migration_tunnelled is enabled this appears
    to be inaccurate from our testing.

    Change-Id: Iff6326f72971364d275ea999418d476007690ef8
    Related-Bug: #1576724
    (cherry picked from commit 71f3fb224a1c4cac0490182543fe20bb8b64d690)

tags: added: in-stable-ussuri
Changed in openstack-ansible:
assignee: Matt Thompson (mattt416) → nobody
assignee: nobody → Andrew Bonney (andrewbonney)
status: Confirmed → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.