neutron

subnet IPs are not available again (not recycled)

Bug #1240353 reported by Roh Tae Won
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Salvatore Orlando
neutron 2013.2 "havana"

Bug Description

In Openstack Havana Neutron, there's a bug with subnet ip availability.

When subnet IP range is exausted(I mean, used all IPs), IP availabilty range information (allocation_pool_id, first_ip, last_ip)
for that subnet is deleted in 'neutron.ipavailabilityranges' table.
Even if you return one of the subnet IP, IP availabilty range information would not created for that subnet on 'neutron.ipavailabilityranges' table.

===================

Senario #1:
  1. make a external network with a subnet of 4 available IPs (no DHCP enabled)
  2. make 4 routers
  3. 'Set Gateway' for 4 routers
  4. 'Clear Gateway' for 1 router
  5. 'Set Gateway' for 1 router results in error 'No more IP address available on network'

Senario #2:
  1. make a network with a subnet of 4 avaialble IPs with DHCP enabled. (ex 192.168.0.0/24, 192.168.0.2~4)
  2. make a #1 VM (that results in private IP with 192.168.0.2, and DHCP will have 192.168.0.3)
  3. make a #2 VM (that results in private IP with 192.168.0.4)
  4. terminate #2 VM
  5. make #3 VM on that network will result in the same error 'No more IP address available on network"

Revision history for this message
Salvatore Orlando (salvatore-orlando) wrote :

Triaging.

Changed in neutron:
assignee: nobody → Salvatore Orlando (salvatore-orlando)
status: New → Confirmed
Revision history for this message
Salvatore Orlando (salvatore-orlando) wrote :

The bug was introduced by the fix for bug 1215350.

PostgreSQL backends will accepts a query with lock for update and an outer join, so we need an inner join.
But when there are no more addresses available the query joining the pool with the availability range will return no records with the effect the the ip allocation is deleted but the address not returned to the pool.

Changed in neutron:
status: Confirmed → Triaged
Salvatore Orlando (salvatore-orlando)
tags: added: db l3-ipam-dhcp
Salvatore Orlando (salvatore-orlando)
Changed in neutron:
importance: Undecided → High
tags: added: havana-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/52050

Changed in neutron:
status: Triaged → In Progress
Thierry Carrez (ttx)
Changed in neutron:
milestone: none → havana-rc3
tags: removed: havana-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/52123

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/52050
Committed: http://github.com/openstack/neutron/commit/72af3cef2a476b7f1f7d109d1bd37cd50a6f560e
Submitter: Jenkins
Branch: master

commit 72af3cef2a476b7f1f7d109d1bd37cd50a6f560e
Author: Salvatore Orlando <email address hidden>
Date: Wed Oct 16 03:14:16 2013 -0700

    Fix IP recycling on exhausted pool

    When recycling an IP, allocation pools are loaded with an inner
    join with ip avaiblity ranges. However, if the pool is exhausted,
    there will be no availability ranges and the query will return
    no results because of the inner join. It will therefore be
    impossible to reuse IP addresses once the pool has been exhausted.

    This patch tries to reload allocation pools without joining on
    IP avaiability ranges if the first query did not return any result.
    Replacing an inner join with an outer join will not be possible
    as postgresql backend does not support 'SELECT ... FOR UPDATE' with
    outer joins (for more info see bug 1215350).

    The patch also contains unit tests for verifying IP are recycled
    correctly even when the pool is exhausted.

    Bug 1240353

    Change-Id: I87f274bd759cb47be77717f0760d3704f9f34ade

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (milestone-proposed)

Reviewed: https://review.openstack.org/52123
Committed: http://github.com/openstack/neutron/commit/9808cac073777ae375074121f9eab15403f1a856
Submitter: Jenkins
Branch: milestone-proposed

commit 9808cac073777ae375074121f9eab15403f1a856
Author: Salvatore Orlando <email address hidden>
Date: Wed Oct 16 03:14:16 2013 -0700

    Fix IP recycling on exhausted pool

    When recycling an IP, allocation pools are loaded with an inner
    join with ip avaiblity ranges. However, if the pool is exhausted,
    there will be no availability ranges and the query will return
    no results because of the inner join. It will therefore be
    impossible to reuse IP addresses once the pool has been exhausted.

    This patch tries to reload allocation pools without joining on
    IP avaiability ranges if the first query did not return any result.
    Replacing an inner join with an outer join will not be possible
    as postgresql backend does not support 'SELECT ... FOR UPDATE' with
    outer joins (for more info see bug 1215350).

    The patch also contains unit tests for verifying IP are recycled
    correctly even when the pool is exhausted.

    Bug 1240353

    Change-Id: I87f274bd759cb47be77717f0760d3704f9f34ade
    (cherry picked from commit 72af3cef2a476b7f1f7d109d1bd37cd50a6f560e)

Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: havana-rc3 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.