MAAS should provide an API to determine available ranges in a Subnet

Landscape team: what APIs do you use to determine which ranges are available?

Several :)

Basically we list the networks and the nodegroups, do the linking between them where they match, for managed networks we subtract the dynamic and static ranges, the broadcast IP, gateway IP and the IP of the nodes.

PS. If MAAS had an API for "tell me what IPs are used on this network" (or
vice versa, IPs are available) we'd gladly use it. Note that the answer we
want is a range, so /ipaddreses/?op=reserve is not sufficient

On Mon, 22 Jun 2015 at 09:30 Adam Collard <email address hidden>
wrote:

> Several :)
>
> Basically we list the networks and the nodegroups, do the linking
> between them where they match, for managed networks we subtract the
> dynamic and static ranges, the broadcast IP, gateway IP and the IP of
> the nodes.
>
> --
> You received this bug notification because you are a member of Landscape
> Bugs, which is subscribed to Landscape Server.
> Matching subscriptions: default
> https://bugs.launchpad.net/bugs/1467120
>
> Title:
> MAAS should consider device addresses as used
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/landscape/+bug/1467120/+subscriptions
>
> --
> Landscape-bugs mailing list
> <email address hidden>
> Modify settings or unsubscribe at:
> https://lists.canonical.com/mailman/listinfo/landscape-bugs
>

After discussion with MAAS team, we agreed that Landscape should query MAAS for the Devices and exclude the IPs they've been allocated from the ones that it (Landscape) considers used.

Can we agree that this is at best a workaround?

It is absolutely a bug in MAAS if it offers up addresses to be used that
it knows are already used.

I have to believe it would be easier to fix this bug in MAAS than to
work around it in Landscape. Only MAAS knows all the different ways an
IP address might be used:

 * attached to an Interface on a Node
 * attached to a MACAddress on a Device
 * associated with a BMC or other controller
 * allocated to a range for some other purpose

There is, or should be, a single function call in MAAS which determines
ranges available. That single function call needs to be fixed to know
about all thee ways an address can be allocated or consumed.

So, to my mind, working around this in Landscape is exactly wrong; it
doubles the total amount of work needed to be done by Canonical to get a
good result (we do it in Landscape and AGAIN in MAAS) and leaves MAAS
buggy in an obvious and important way.

Please reconsider. If needed, have someone on the Landscape team submit
the fix to MAAS.

Mark

Hi Mark,

We agree with you here.

The new subnet's API in 1.9 will support the determination of the available ranges per subnet. This is targeted to be fixed by 1.9 and as part of the new networking features.

The work around mentioned above, however, was only meant to be used against 1.8 and not 1.9.

Hope this helps.

Note that simply determining the available ranges isn't enough. Landscape needs to be able to reserve an IP range in MAAS. Otherwise someone could assign a device address to an IP that Landscape already given to OpenStack.

Yes, MAAS should offer APIs to register devices / containers, and whole
ranges.

Going forward, we can unify the "static" and "DHCP" ranges per subnet
with this idea; these just become classes of range reserved "by MAAS for
MAAS", while things like Landscape can reserve ranges for their own
purposes.

The tricky part will be in handling the teardown and scaling of such
ranges, i.e. how to modify the DHCP range once there are already DHCP
leases in place.

Mark

Here are my current thoughts on what this may look like in MAAS 1.9; feedback appreciated. This isn't set in stone, as I'm still iterating with the team. But stakeholder feedback would be great.

Use cases (2) and (3), I think, are optional. But (3) would be nice to have for support/troubleshooting purposes.

I added the address_family parameter because I don't think this makes a lot of sense for IPv6. And on second thought, reserving a typical IPv6 range could entail adding a few billion addresses to the database in the current design, so that won't work. (So maybe we should make these APIs IPv4-only for now; with the possible exception of "find_reserved_ip_addresses".)

Use cases:

(1) As a MAAS API user, I want to determine which IP ranges within a specific MAAS-managed subnet are free.

# maas <profile> subnet find_unused_ip_ranges <subnet_id> [address_family=<all|ipv4|ipv6>]
[
    {
        "start": "10.0.0.2"
        "end": "10.0.0.49"
        "num_addresses": 48
    },
    {
        "start": "10.0.0.151"
        "end": "10.0.0.253"
        "num_addresses": 103
    }
]

(2) As a MAAS API user, I want to determine which IP ranges are unused across the system.

# maas <profile> subnets find_unused_ip_ranges [address_family=<all|ipv4|ipv6>]
[
    {
        "subnet": <Subnet>,
        "management": <unmanaged|dhcp|dhcp-and-dns>,
        "ranges": [<see-above>]
    }
]

(3) As a MAAS API user, I want to determine which IP addresses are in use for a particular subnet, and why.

# maas <profile> subnet find_reserved_ip_addresses <subnet_id> [address_family=<all|ipv4|ipv6>]
[
    {
        "start": "10.0.0.1",
        "end": "10.0.0.1",
        "num_addresses": 1,
        "reason": "gateway"
    },
    {
        "start": "10.0.0.50",
        "end": "10.0.0.99",
        "num_addresses": 49,
        "reason": "maas-managed-dynamic"
    },
    {
        "start": "10.0.0.100",
        "end": "10.0.0.150",
        "num_addresses": 49,
        "reason": "maas-managed-static"
    },
    {
        "start": "10.0.0.254",
        "end": "10.0.0.254",
        "num_addresses": 1,
        "reason": "dns"
    },
]

(4) As a MAAS API user, I want to reserve a range of IP addresses.

# maas <profile> subnet reserve_ip_range <subnet_id> start=<start_address> end=<end_address> num_addresses=<num_addresses>

Note: there is some discussion about modeling IP ranges as first-class citizens, so that we can support IPv6, and easily *un-reserving* a range. Without modeling ranges, to undo the command in (4) above would require num_addresses calls to the `release` API, which isn't good.

There is debate about whether or not this fits into 1.9, but it looks like we'll have to do it in order to do this the right way.

This may also help as we convert the concepts of dynamic and static ranges attached to a cluster interface to the more generic concept of DHCP-reserved ranges, and ranges reserved for some other reason.

Mike, I think we figured out that we only need a more limited set of
APIs. Basically, rather than provide the user with all the ranges, just
let them ask for:

 * the total number of available addresses in a space / subnet
 * the largest number of contiguous addresses

If Launchpad needs 5 addresses, it can ask "what's the largest number of
addresses" and if it's more than 5 it can then say:

 * request a range of n addresses for a purpose (text)
    - it gets a range with an id (token unique to that reservation)
    - maas reserves the range for (user) (purpose)

 * release the range with a specific token
    - maas releases the range

In other words, rather than giving the consumer all the details and
letting them choose their own ranges, treat addresses as dynamic
resources ("I need 5 in this space") rather than things to be managed
with a lot of insight. Another way to look at it is this: we do not want
Landscape having to have code to decide WHICH set of 5 addresses to use;
they should only care AT MOST that they get it on a particular space, or
subnet (but space is better).

Make sense?

Mark

On 09/09/15 05:33, Mike Pontillo wrote:
> Note: there is some discussion about modeling IP ranges as first-class
> citizens, so that we can support IPv6, and easily *un-reserving* a
> range. Without modeling ranges, to undo the command in (4) above would
> require num_addresses calls to the `release` API, which isn't good.

EXACTLY!

The whole reservation is "one range with one ID and one purpose for one
user".

Within that reservation, MAAS might in future observe that some
addresses are being used (visibly passing traffic) and others are not.
But at the end of the day, it has allocated that block of addresses as a
single entity to that user for a given purpose (text field that
Landscape or a human could type text in, mainly to help others find out
who to talk to).

> There is debate about whether or not this fits into 1.9, but it looks
> like we'll have to do it in order to do this the right way.

Yes, I think we're just papering over if we do it address by address.

> This may also help as we convert the concepts of dynamic and static
> ranges attached to a cluster interface to the more generic concept of
> DHCP-reserved ranges, and ranges reserved for some other reason.

Yes, exactly that :) In other words, think of a DHCP range as just "a
range MAAS allocated to itself for the purpose of DHCP".

So say I am an admin user. I just go and say "I want another range of 10
addresses for DHCP". This shows up as:

 start - end mark maas-dhcp

In a future version I can come along and release the range (though MAAS
would hold on to it till all the leases expire to be pedantic :)). But
certainly, Landscape could release a range it had reserved.

Mark

At this point, I have landed [in MAAS 1.9] APIs to view reserved (or unreserved) APIs in a particular subnet; basically what was described in (1) and (3) above, but see the MP for example output.[1]

In summary, the change I landed provides a good foundation to address the remainder of the user stories now under discussion. But it does not address the specific use cases mentioned in the follow-on comments. I've filed two additional bugs to track those against; see below. Let's continue the discussion on those new bugs, as needed.

I had already planned for another user story to land separately in 1.9:

 - As an API user (and on the web UI), I want to see the following details about a subnet:
   * number of used IP addresses
   * number allocated IP addresses
   * percentage utilization
   * largest contiguous block of free addresses

It sounds like it would be useful to have the same information available on Space (as an aggregation of the same information for all the subnets), so per your comments, s/subnet/subnet or space/ above.

I have filed bug #1498221 to track this.[2]

I was uncertain about whether or not we wanted to address the following user story in 1.9, (pending implementation of IP ranges) but it sounds like we need it for Landscape's convenience either way:

 - As a MAAS administrator, I want to ask MAAS to reserve N arbitrary addresses in a Subnet or Space.
   * Note: for this operation, it seems like MAAS should prefer *smaller* blocks of free address space, in case larger blocks are needed for other purposes.

I've filed bug #1498224 to track this.[3]

For maas-next we can model IP ranges properly, so that they can be easily released once claimed. (this will require refactoring the way IP addresses are managed across the system, impact the way cluster interfaces are defined, etc)

[1]: https://code.launchpad.net/~mpontillo/maas/subnet-ranges-apis/+merge/271577

[2]: https://bugs.launchpad.net/maas/+bug/1498221

[3]: https://bugs.launchpad.net/maas/+bug/1498224