StarlingX R2 duplex: VM not getting IP assigned to the vlan network when re-spawned on controller-0 after shutting down controller-1

Assigning to the networking team to triage/investigate. This issue is not reported for any stx.2.0 testing done by the verification teams. It may be a procedural issue.

Hi Akshay,

Could you please provide the log when VM gets rebulid on controller-0? This bug is similar to the previous one. I suspect that the vlan dhcp-agent hasn`t been reconstructed on controller-0.

Hi Marvin,

Which logs ?
1. Are you asking about the "nova-conductor.log" or nova components related logs ?
   -If yes, I can't find any logs in any nova containers. I can't even enable it in configuration files as it
    states "Changing a read-only file" and "Can't open file for writing".
2. or you are asking for >collect all ?
   -If yes, then for which controller node?

Hi Akshay,

It`s ok, I just want to see instance console log. I have tried to reproduce this bug, but I've run into some problems now. could you please provide more detailed steps? e.g. steps to create ipv6 network, steps to create vm.

I also have several questions as below:
1. When the VM is created, does the VM get IPv6 address inside ?
2. Which controller node does the flat VM run on?

Sure Marvin,

Answers to your queries:
1. Yes, it gets IPv6 address assigned inside the VM initially before HA.
2. Flat IPv6 and vlan IPv6 both networks are attached on VMs, Initially both VMs have IPs assigned on both the networks ON EACH CONTROLLER NODE.

Steps tp create IPv6 networks:
1. I have created both the networks from horizon with abcd:abcd:abcd::0/64 as subnet and IPv6 as network type.
2. For flat_IPv6, physical network: physnet0
3. For vlan_IPv6, physical network: physnet1

Steps to create VM:
1. Image:centos
2. Networks: flat_IPv6, vlan_IPv6
3. No Security group added.
4. Cloud Config used only to change login credentials as username:centos and passwd:XXXXXX.

HI Marvin,

One new finding for your information !

When I tried the above same test case WITH VOLUMES ATTACHED to VMs, then I didn't face this VLAN IP issue. Respawned VM gets IPs assigned as expected.

Hi Akshay,
Thanks for the info. This seems to be an occasional bug, I'll keep an eye on it. If you have any new discoveries, please let me know. thanks a lot~

Hi Marvin,

I have tested both the cases (with volume and without volume) many times and it is reproducible every time for me. So I don't think it is an occasional bug.

Hi Akshay,

which ipv6-address-mode did you specify?

and could you please provide the openstack image file?
Thanks a lot.

Hi Team,

I am creating 2 VM simultaneously on VLAN IPv4 network using cirros image.
Out of the 2 VM, one VM gets spawned on Controller-0 and other on Controller-1.

The VM which gets spawned on Controller-1 is failing the lease of getting IP as a result of which there is no IP on the VM interface, which gets spawned on Controller-1.

The Controller-0 VM successfully comes up.

I am attaching "collect" logs of both the controllers for your reference.

The network configuration can be configured via cloud-init, but for IPv6 the metadata service is not available. Therefore you have to use the config-drive=true when booting a nova instance.

Manual network configuration, you can specify additional configuration via cloud-init. This will require a #cloud-config network entry. For the latest ubuntu images, this would use netplan to configure the static addressing.

@Matt I just take over this bug. Could you share more details of the network configuration? I am not familiar with cloud-init.

@Akshay, which centos image you used? Could you point out me where to download it?

Hi Marvin,

I did not specify any IPv6 mode. I used "Not specified (default)" option.

Based on review in the starlingx networking team on 2020-02-20, we believe this is a configuration/setup issue. There is nothing in the starlingx software that would result in this issue. See comment above from Matt Peters:
https://bugs.launchpad.net/starlingx/+bug/1849655/comments/12

Regarding IPv6, the initial issue was reported for IPv6.
Steps to Reproduce:
2. Spawn 2 VMs together on IPv6 flat and IPv6 vlan from horizon.

Will leave open for reporter to provide more information, but lowering the priority.