Fuel for OpenStack

Check whether network is correct before deployment start

Bug #1453662 reported by Sam Stoelinga
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Medium
Matthew Mosesohn
Fuel for OpenStack 8.0
6.0.x
Invalid
High
Fuel Python (Deprecated)
Fuel for OpenStack 6.0-updates
7.0.x
Invalid
Medium
Fuel Library (Deprecated)
Fuel for OpenStack 7.0-updates

Bug Description

Steps to reproduce:
1. Deploy a 6 node cluster: (3 node controller. 3 node compute with Ceph)
2. After environment was created deployed, delete the 3 compute + ceph nodes from cluster and press deploy, wait for it to finish
3. Re-add the 3 compute nodes again as compute + ceph. Press deploy

Current result:
The controller gets stuck at Installing OpenStack process with fails see below error message

Expected result:
The addition of the 3 nodes should succeed without errors.

Error messsage found in puppet:

2015-05-11 14:10:23 ERR

 (/Stage[main]/Openstack::Controller/Openstack::Network::Create_network[net04_ext]/Neutron_subnet[net04_ext__subnet]/cidr) change from 192.168.15.0/24 to 192.168.15.131/24 failed: Property cidr does not support being updated

2015-05-11 14:10:23 ERR

 /usr/bin/puppet:4

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `benchmark'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `apply_catalog'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `apply'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `evaluate'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util.rb:327:in `thinmark'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `evaluate'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `call'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:187:in `eval_resource'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:18:in `evaluate'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `each'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:82:in `perform_changes'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:130:in `sync_if_needed'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:193:in `sync'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/property.rb:581:in `sync'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/property.rb:503:in `set'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/property.rb:178:in `call_provider'

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/property.rb:178:in `send'

2015-05-11 14:10:23 ERR

 /etc/puppet/modules/neutron/lib/puppet/provider/neutron_subnet/neutron.rb:216:in `cidr='

2015-05-11 14:10:23 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:97:in `fail'

2015-05-11 14:10:23 ERR

 Property cidr does not support being updated

2015-05-11 14:10:23 NOTICE

 (/Stage[main]/Openstack::Controller/Openstack::Network::Create_network[net04_ext]/Notify[net04_ext ::: network_type local]/message) defined 'message' as 'net04_ext ::: network_type local'

2015-05-11 14:10:23 NOTICE

 net04_ext ::: network_type local

2015-05-11 14:10:23 NOTICE

 (/Stage[main]/Openstack::Controller/Openstack::Network::Create_network[net04]/Notify[net04 ::: shared false]/message) defined 'message' as 'net04 ::: shared false'

2015-05-11 14:10:23 NOTICE

 net04 ::: shared false

2015-05-11 14:10:23 NOTICE

 (/Stage[main]/Openstack::Controller/Openstack::Network::Create_network[net04]/Notify[net04 ::: tenant admin]/message) defined 'message' as 'net04 ::: tenant admin'

2015-05-11 14:10:23 NOTICE

 net04 ::: tenant admin

2015-05-11 14:10:11 NOTICE

 (/Stage[main]/Openstack::Controller/Openstack::Network::Create_network[net04]/Notify[net04 ::: network_type vlan]/message) defined 'message' as 'net04 ::: network_type vlan'

2015-05-11 14:10:11 NOTICE

 net04 ::: network_type vlan

2015-05-11 14:10:10 NOTICE

 (/Stage[main]/Openstack::Controller/Openstack::Network::Create_network[net04]/Notify[net04 ::: physnet physnet2]/message) defined 'message' as 'net04 ::: physnet physnet2'

2015-05-11 14:10:10 NOTICE

 net04 ::: physnet physnet2

2015-05-11 14:10:10 NOTICE

 (/Stage[main]/Openstack::Controller/Openstack::Network::Create_network[net04]/Notify[net04 ::: router_ext false]/message) defined 'message' as 'net04 ::: router_ext false'

2015-05-11 14:10:10 NOTICE

 net04 ::: router_ext false

2015-05-11 14:10:08 NOTICE

 (/Stage[main]/Openstack::Network/Exec[waiting-for-neutron-api]/returns) executed successfully

2015-05-11 14:10:07 NOTICE

 (/Stage[main]/Neutron::Server/Service[neutron-server]) Triggered 'refresh' from 1 events

2015-05-11 14:10:00 NOTICE

 (/Stage[main]/Cluster::Neutron::Dhcp/Cluster::Corosync::Cs_service[dhcp]/Tweaks::Ubuntu_service_override[neutron-dhcp-agent]/File[create_neutron-dhcp-agent_override]/ensure) created

2015-05-11 14:10:00 NOTICE

 (/Stage[main]/Openstack::Controller/Tweaks::Ubuntu_service_override[nova-cert]/Exec[remove_nova-cert_override]/returns) executed successfully

2015-05-11 14:09:59 NOTICE

 (/Stage[main]/Openstack::Ha::Heat/Openstack::Ha::Haproxy_service[heat-api-cfn]/Exec[haproxy reload for heat-api-cfn]/returns) executed successfully

2015-05-11 14:09:59 NOTICE

 (/Stage[main]/Openstack::Ha::Heat/Openstack::Ha::Haproxy_service[heat-api-cfn]/Exec[haproxy reload for heat-api-cfn]/returns) 2015/05/11_14:09:57 INFO: [WARNING] 130/140957 (19778) : config : 'stats' statement ignored for proxy 'rabbitmq' as it requires HTTP mode. [WARNING] 130/140957 (19778) : config : 'stats' statement ignored for proxy 'mysqld' as it requires HTTP mode. [WARNING] 130/140957 (19778) : config : 'stats' statement ignored for proxy 'murano_rabbitmq' as it requires HTTP mode.

Sam Stoelinga (sammiestoel)
summary: - Redeploying compute node fails
+ Redeploying compute node fails at puppet provider network
Nastya Urlapova (aurlapova)
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
status: New → Incomplete
Revision history for this message
Bogdan Dobrelya (bogdando) wrote : Re: Redeploying compute node fails at puppet provider network

Please provide a diagnostic logs snapshot and describe the environment settings used

Changed in fuel:
importance: Undecided → High
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

Sam, thanks for your report, could you attach diagnostic snapshot and fuel version?

Bogdan Dobrelya (bogdando)
Changed in fuel:
milestone: none → 6.1
Revision history for this message
Sam Stoelinga (sammiestoel) wrote :

found in latest fuel 6.0 iso from official download page. Will attach snapshot after it's uploaded to external site. It already became 400 MB now.

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

472 iso, can not reproduce

Revision history for this message
Nastya Urlapova (aurlapova) wrote :

marked as Invalid for 6.1

Changed in fuel:
status: Incomplete → Invalid
Revision history for this message
Leontiy Istomin (listomin) wrote :

reproduced with 6.1-521 build:

1. try to deploy the following environment:
Baremetal,Centos,IBP,HA, Neutron-gre,Ceph-all,Nova-debug,Nova-quotas, 6.1-521
Controllers:3 Computes:47
2. deployment was failed due some Ceph bug
3. changed puppet manifest and click "deploy changes" button.
4. Deployment was failed with the following error (node-48 is primary controller):
Deployment has failed. Method granular_deploy. Deployment failed on nodes 48. Inspect Astute logs for the details
From puppet logs of node-48:
http://paste.openstack.org/show/294466/

As a workaround you can delete interface of neutron router "router04" which connected to internal network "net04":
http://paste.openstack.org/show/294467/
And then click "deploy changes" again. Deployment will be successful.

Changed in fuel:
status: Invalid → Confirmed
tags: added: scale
Revision history for this message
Leontiy Istomin (listomin) wrote :

for workaround in #6 comment we also need to delete internal subnetwork og net04 network before press "deploy changes" last time:
http://paste.openstack.org/show/294489/

Stanislav Makar (smakar)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Stanislav Makar (smakar)
Revision history for this message
Stanislav Makar (smakar) wrote :

Fuel tasks were not synced after fuel-library update

Revision history for this message
Stanislav Makar (smakar) wrote :

synced and we are re deploying, waiting for results

Bogdan Dobrelya (bogdando)
tags: added: l23network
Sergey Vasilenko (xenolog)
tags: added: neutron
removed: l23network
Revision history for this message
Stanislav Makar (smakar) wrote :

did not reproduce again on the same lab

Changed in fuel:
status: Confirmed → Invalid
Revision history for this message
Stanislav Makar (smakar) wrote :

there was the problem with the way of applying patches

Revision history for this message
Stanislav Makar (smakar) wrote :

Error: /Stage[main]/Main/Openstack::Network::Create_network[net04]/Neutron_subnet[net04__subnet]/cidr: change from 192.168.108.0/22 to 192.168.111.0/22 failed: Property cidr does not support being updated

admin network was changed
puppet want to changes network

Revision history for this message
Stanislav Makar (smakar) wrote :

root@node-13:~# neutron subnet-show 8f1f2e83-9c6d-4409-9c77-b4b352a20cf8
+-------------------+------------------------------------------------------+
| Field | Value |
 +-------------------+------------------------------------------------------+
| allocation_pools | {"start": "192.168.108.1", "end": "192.168.111.0"} |
| | {"start": "192.168.111.2", "end": "192.168.111.254"} |
| cidr | 192.168.108.0/22 |
| dns_nameservers | 8.8.4.4 |
| | 8.8.8.8 |
| enable_dhcp | True |
| gateway_ip | 192.168.111.1 |
| host_routes | |
| id | 8f1f2e83-9c6d-4409-9c77-b4b352a20cf8 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | net04__subnet |
| network_id | 4383540f-4f34-44fd-99f8-e3d863383f04 |
| tenant_id | 5d224c475500423f9d2eee4003f38fae |
 +-------------------+------------------------------------------------------+

it is misconfiguration
network 192.168.111.0/22 could not be due to it is included in 192.168.108.0/22

Network: 192.168.108.0/22 (Class C)
Netmask: 255.255.252.0 = 22
Wildcard: 0.0.3.255
Broadcast: 192.168.111.255
HostMin: 192.168.108.1
HostMax: 192.168.111.254

First time Neutron fixes by itself but during next redeploy we are trying to set again 192.168.111.0/22 -- it fails

Revision history for this message
Stanislav Makar (smakar) wrote :

We should check correct network in nailgun and fail deployment if it is bad or even do not start it

summary: - Redeploying compute node fails at puppet provider network
+ Check whether network is correct before deployment start
Revision history for this message
Sam Stoelinga (sammiestoel) wrote :

Stanlisas the issue I found seems to be different from what you're experiencing. I did not make any changes to neutron subnet I just used default so my network was correct before the deployment start.

Suggest filing a new bug and changing summary back to original. I may not have time to follow this closely though.

Revision history for this message
Vladimir Sharshov (vsharshov) wrote :

Looks like this bug covered by https://blueprints.launchpad.net/fuel/+spec/advanced-networking. Moving to 8.0

Revision history for this message
Vladimir Sharshov (vsharshov) wrote :

Correct link to blueprint: https://blueprints.launchpad.net/fuel/+spec/granular-network-functions

Changed in fuel:
milestone: 7.0 → 8.0
Roman Rufanov (rrufanov)
tags: added: customer-found support
Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/8.0.x
Vladimir Sharshov (vsharshov)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Fuel Library Team (fuel-library)
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

 (/Stage[main]/Openstack::Controller/Openstack::Network::Create_network[net04_ext]/Neutron_subnet[net04_ext__subnet]/cidr) change from 192.168.15.0/24 to 192.168.15.131/24 failed: Property cidr does not support being updated
We should either fix the provider to evaluate a real_cidr that is a network address (and not a specific IP + cidr notation) or generate a clean address to send the provider

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Matthew Mosesohn (raytrac3r)
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

How is it possible to change example admin network (net04) after deployment? Or did a user change it manually?

Revision history for this message
Vasily Gorin (vgorin) wrote :

It doesn't reproduce on build 140 8.0-kilo

Changed in fuel:
status: Confirmed → Invalid
Dmitry Pyzhov (dpyzhov)
tags: added: area-library
Leontiy Istomin (listomin)
tags: removed: scale
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.