Creating floating IP range with nova-network fails due to missing admin role for admin user

Bug #1407675 reported by Andrey Sledzinskiy
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Vladimir Kuklin
5.0.x
Fix Committed
High
Bogdan Dobrelya
5.1.x
Fix Committed
High
Bogdan Dobrelya
6.0.x
Fix Committed
High
Bartłomiej Piotrowski
6.1.x
Fix Released
High
Vladimir Kuklin

Bug Description

{

    "build_id": "2015-01-02_22-54-45",
    "ostf_sha": "249574cdda0279dc8ec4957a5979651439476e8a",
    "build_number": "40",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "4b325a95b0217a26a17f526cb734b3748cb03e12",
    "production": "docker",
    "fuelmain_sha": "cc47eef01622b8fdf2d8f290cb8dfb46738dc7f5",
    "astute_sha": "18be5cd3b819f3cad4c970ce5f72d3fb211a0969",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.1",
    "release_versions": {
        "2014.2-6.0": {
            "VERSION": {
                "build_id": "2015-01-02_22-54-45",
                "ostf_sha": "249574cdda0279dc8ec4957a5979651439476e8a",
                "build_number": "40",
                "api": "1.0",
                "nailgun_sha": "4b325a95b0217a26a17f526cb734b3748cb03e12",
                "production": "docker",
                "fuelmain_sha": "cc47eef01622b8fdf2d8f290cb8dfb46738dc7f5",
                "astute_sha": "18be5cd3b819f3cad4c970ce5f72d3fb211a0969",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.1",
                "fuellib_sha": "42df19509c40e2cdc9ede9d89b42188ea27c1b7e"
            }
        }
    },
    "fuellib_sha": "42df19509c40e2cdc9ede9d89b42188ea27c1b7e"

}

Steps:
1. Create next cluster - Simple, CentOS, flat nova-network, 1 controller, 1 compute
2. Run deployment of the cluster

Actual result - deployment got stuck on 57% progress with errors in puppet log (node-2):

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /etc/puppet/modules/nova/lib/puppet/provider/nova_floating_range/nova_manage.rb:15:in `exists?'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /etc/puppet/modules/nova/lib/puppet/provider/nova_floating_range/nova_manage.rb:49:in `operate_range'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/compute/connection.rb:476:in `get_floating_ips_bulk'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/connection.rb:207:in `req'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/connection.rb:512:in `raise_exception'

2015-01-03 02:27:24 ERR

 (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) Could not evaluate: The server returned status 403

Logs are attached

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Revision history for this message
Aleksey Kasatkin (alekseyk-ru) wrote :
Download full text (3.5 KiB)

Successfully deployed on

{"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}}, "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}

I took your network configuration:

2015-01-03 02:14:08.486 DEBUG [7fe67715f740] (logger) Request PUT /api/clusters/1/network_configuration/nova_network from 172.17.42.1:44197 {"networking_parameters": {"dns_nameservers": ["8.8.4.4", "8.8.8.8"], "net_manager": "FlatDHCPManager", "fixed_networks_vlan_start": 103, "fixed_networks_amount": 1, "floating_ranges": [["10.108.31.128", "10.108.31.254"]], "fixed_network_size": 256, "fixed_networks_cidr": "10.1.0.0/24"}, "networks": [{"name": "fixed", "ip_ranges": [], "gateway": null, "meta": {"ext_net_data": ["fixed_networks_vlan_start", "fixed_networks_amount"], "name": "fixed", "notation": null, "render_type": null, "assign_vip": false, "map_priority": 2, "use_gateway": false, "vlan_start": null, "render_addr_mask": null, "configurable": false}, "vlan_start": null, "cidr": null, "group_id": 1, "id": 5}, {"name": "public", "ip_ranges": [["10.108.31.2", "10.108.31.127"]], "gateway": "10.108.31.1", "meta": {"name": "public", "notation": "ip_ranges", "render_type": null, "assign_vip": true, "map_priority": 1, "use_gateway": true, "vlan_start": null, "render_addr_mask": "public", "cidr": "172.16.0.0/24", "configurable": true, "gateway": "172.16.0.1", "ip_range": ["172.16.0.2", "172.16.0.127"]}, "vlan_start": null, "cidr": "10.108.31.0/24", "group_id": 1, "id": 2}, {"name": "management", "ip_ranges": [["10.108.32.2", "10.108.32.254"]], "gateway": "10.108.32.1", "meta": {"name": "management", "notation": "cidr", "render_type": "cidr", "assign_vip": true, "map_priority": 2, "use_gateway": false, "vlan_start": 101, "render_addr_mask": "internal", "cidr": "192.168.0.0/24", "configurable": true}, "vlan_start": null, "cidr": "10.108.32.0/24", "group_id": 1, "id": 3}, {"name": "storage", "ip_ranges": [["10.108.34.2", "10.108.34.254"]], "gateway": "10.108.34.1", "meta": {"name": "storage", "notation": "cidr", "render_type": "cidr", "assign_vip": false, "map_priority": 2, "use_gateway": false, "vlan_start": 102, "render_addr_mask": "storage", "cidr": "192.168.1.0/24", "configurable": true}, "vlan_start": null, "cidr": "10.108.34.0/24", "group_id": 1, "id": 4}, {"name": "fuelweb_admin", "ip_ranges": [["10.108.30.3", "10.108.30.254"]], "gateway": null, "meta":...

Read more...

Revision history for this message
Aleksey Kasatkin (alekseyk-ru) wrote :
Ryan Moe (rmoe)
Changed in fuel:
status: New → Confirmed
Revision history for this message
Ryan Moe (rmoe) wrote :

nova-api returned a 403 because compute_extension:floating_ips_bulk requires the admin role. The novaSimpleFlat user (admin user from settings tab) did not have the admin role added until 7 seconds after the attempt to configure the floating IPs.

2015-01-03T02:27:22.950945+00:00 info: (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) Starting to evaluate the resource

2015-01-03T02:27:29.742637+00:00 notice: (/Stage[main]/Keystone::Roles::Admin/Keystone_user_role[novaSimpleFlat@novaSimpleFlat]/roles) roles changed ['_member_'] to 'admin'

Changed in fuel:
status: Confirmed → Triaged
tags: added: low-hanging-fruit
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/152674

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Ryan Moe (rmoe)
status: Triaged → In Progress
Revision history for this message
Stanislaw Bogatkin (sbogatkin) wrote : Re: Deployment of simple centos cluster failed with (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) Could not evaluate: The server returned status 403

We don't support simple mode in 6.1 anymore.

Revision history for this message
Ryan Moe (rmoe) wrote :

The way to reproduce this in simple mode is simply by changing the admin username to something besides admin. This causes things to run out of order (I don't understand why) and fail because there is no dependency (in both simple and HA modes) between the admin role and floating ip pool creation.

So I have two questions:

1) Is simple mode going to be removed completely or will it stay in experimental mode as it now?
2) Is it possible that a similar small and seemingly unrelated change could cause the same failure in HA mode?

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

The simple mode was deprecated for 6.0 and removed in 6.1. We should not consider the issue with deprecated deployment mode as a high one

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/156180

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/156181

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/152674
Reason: Chis change is abandoned as related bug is won't fix.

Changed in fuel:
status: In Progress → Won't Fix
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.1)

Reviewed: https://review.openstack.org/156181
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=1fb61716e1d8975e267342614044b9892319cfd2
Submitter: Jenkins
Branch: stable/5.1

commit 1fb61716e1d8975e267342614044b9892319cfd2
Author: Ryan Moe <email address hidden>
Date: Tue Feb 3 12:36:58 2015 -0800

    Ensure admin role is assigned prior to creating floating range

    Nova API will return a 403 if a non-admin user attempts to create
    a floating IP range.

    Change-Id: I60ad318ba3bb91c863d58755f2b03e73099efc2b
    Closes-bug: #1407675
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.0)

Reviewed: https://review.openstack.org/156180
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=25bdfff6f25c32204ddf83dcf392ced3dd4743df
Submitter: Jenkins
Branch: stable/5.0

commit 25bdfff6f25c32204ddf83dcf392ced3dd4743df
Author: Ryan Moe <email address hidden>
Date: Tue Feb 3 12:36:58 2015 -0800

    Ensure admin role is assigned prior to creating floating range

    Nova API will return a 403 if a non-admin user attempts to create
    a floating IP range.

    Change-Id: I60ad318ba3bb91c863d58755f2b03e73099efc2b
    Closes-bug: #1407675
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message
Ryan Moe (rmoe) wrote : Re: Deployment of simple centos cluster failed with (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) Could not evaluate: The server returned status 403

Moving to confirmed for 6.1 and 6.0 as this bug was reproduced in an HA deployment here: https://bugs.launchpad.net/fuel/+bug/1427194

Changed in fuel:
status: Won't Fix → Confirmed
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.0)

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/161701

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/6.0)

Reviewed: https://review.openstack.org/161701
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=591d5d51ebccb07df6ec8030b9b664347bcf71f0
Submitter: Jenkins
Branch: stable/6.0

commit 591d5d51ebccb07df6ec8030b9b664347bcf71f0
Author: Ryan Moe <email address hidden>
Date: Tue Feb 3 12:36:58 2015 -0800

    Ensure admin role is assigned prior to creating floating range

    Nova API will return a 403 if a non-admin user attempts to create
    a floating IP range.

    Change-Id: I60ad318ba3bb91c863d58755f2b03e73099efc2b
    Closes-bug: #1407675
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 1fb61716e1d8975e267342614044b9892319cfd2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/162647

Changed in fuel:
assignee: Ryan Moe (rmoe) → Vladimir Kuklin (vkuklin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/152674
Reason: The fix for master branch superseded by https://review.openstack.org/162647

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/162647
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=62e68af896887ebe18944e6a0a9721e269119ad4
Submitter: Jenkins
Branch: master

commit 62e68af896887ebe18944e6a0a9721e269119ad4
Author: Vladimir Kuklin <email address hidden>
Date: Mon Mar 9 17:53:08 2015 +0300

    Ensure admin role is assigned prior to creating floating range

    Nova API will return a 403 if a non-admin user attempts to create
    a floating IP range.

    Change-Id: I5458f9e261ecf90554db264f7b3d3a8b4f0ce468
    Closes-bug: #1407675

Changed in fuel:
status: In Progress → Fix Committed
Ryan Moe (rmoe)
summary: - Deployment of simple centos cluster failed with
- (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254])
- Could not evaluate: The server returned status 403
+ Creating floating IP range with nova-network fails due to missing admin
+ role for admin user
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified on ISO #216 for non-default and default admin username. Both deployments were successful

"build_id": "2015-03-22_22-54-44", "ostf_sha": "b4d284e9364e30bf5162975c2ba0be6ca0f14ebd", "build_number": "216", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-22_22-54-44", "ostf_sha": "b4d284e9364e30bf5162975c2ba0be6ca0f14ebd", "build_number": "216", "api": "1.0", "nailgun_sha": "51974b50c3961be3ed0fdc7859570db2eeb83e9c", "production": "docker", "python-fuelclient_sha": "b223dcaf5fdad2f714cd245958fefe03995d6207", "astute_sha": "4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "f52e4442df55a2b62637a2cf4038a24ba6f37b6f", "fuellib_sha": "a636c680e3c7d8cc66ed3e03645f38250beb8970"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "51974b50c3961be3ed0fdc7859570db2eeb83e9c", "production": "docker", "python-fuelclient_sha": "b223dcaf5fdad2f714cd245958fefe03995d6207", "astute_sha": "4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "f52e4442df55a2b62637a2cf4038a24ba6f37b6f", "fuellib_sha": "a636c680e3c7d8cc66ed3e03645f38250beb8970"

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.