observed IP is not released if dynamic range is modified before lease expire
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Triaged
|
Medium
|
Unassigned |
Bug Description
Normal behavior:
When commissioning a machine, IPs from the reserved dynamic range are used. During commissioning, the IPs are listed in the subnet page as "Observed". After commissioning is complete, the IPs remain in the list for 10 minutes (default lease time) until the lease expires, and then it is released back to the reserved dynamic range pool.
Bug:
If, at any time between commissioning has started and the lease expires, the reserved dynamic range is modified to not include the dynamic IP used by the machine, then the IP is no longer released when the lease expires. The IP then stays indefinitely listed as "Observed" in the subnet list and cannot be used anywhere else (example: since it is no longer in the dynamic range, it should be usable as a static IP), because MaaS wouldn't allow it saying the IP is already in-use.
It has been validated that the problem persists even with machine turned off with "Ready" status and waiting 10 minutes.
How it should behave:
The IP should get released when its lease expires since it was allocated from the dynamic range. Therefore it would not be shown in the subnet list anymore and could be used as a static IP afterwards.
Ideally, the modification to the reserved dynamic range should not cause the issue.
Alternatively, modification of the dynamic reserved range could be not allowed while there are IPs waiting on to have their leases expired.
Workaround:
The only way to address the problem is either re-commissioning the machine, or deleting it. Then the IP gets removed from the list and freed so it can be re-used.
The workaround is acceptable if the machine's role is not important, but it is very unpleasant if it is part of a production cloud.
Tested on MaaS versions:
- 2.7.3-8291-
- 2.9.0~beta2-
Steps to reproduce:
1) Commission the machine
2) Wait for the machine to obtain dynamic IP or be successfully commissioned
3) Within 10 minutes after commissioning has started/completed, change the dynamic range to not include the selected IP
4) Wait 10 minutes (since machine commissioning is complete) for lease to expire
5) IP remains instead of being released
description: | updated |
tags: | added: sts |
description: | updated |
tags: | added: se-00292194 |
Changed in maas: | |
milestone: | 3.4.0 → 3.4.x |
Investigating further into this, part of the reason the bug happens is because MaaS keeps track of the leases in the file /var/lib/ maas/dhcp/ dhcpd.leases which is read by the dhcpd process.
In that file there are instructions for what to run when the lease expires, such as:
lease 192.168.105.222 { class-identifie r = "Linux ipconfig";
binary- to-ascii (16, 8, ":",
substring (hardware, 1, 6)) ;
binary- to-ascii (10, 8, ".", leased-address) ; sbin/maas- dhcp-helper" , "notify", "--action", "expiry", maas/dhcpd. sock");
binary- to-ascii (16, 8, ":",
substring (hardware, 1, 6)) ;
binary- to-ascii (10, 8, ".", leased-address) ; sbin/maas- dhcp-helper" , "notify", "--action", "release", maas/dhcpd. sock");
starts 5 2020/09/25 19:30:28;
ends 5 2020/09/25 19:40:28;
cltt 5 2020/09/25 19:30:28;
binding state active;
next binding state free;
rewind binding state free;
hardware ethernet 52:54:00:13:2c:8d;
set clht = "ubuntu";
set cllt = "600";
set clip = "192.168.105.222";
set clhw = "52:54:0:13:2c:8d";
set vendor-
client-hostname "ubuntu";
on expiry {
set clhw =
set clip =
execute ("/usr/
"--mac", clhw, "--ip-family", "ipv4", "--ip", clip, "--socket", "/var/lib/
}
on release {
set clhw =
set clip =
execute ("/usr/
"--mac", clhw, "--ip-family", "ipv4", "--ip", clip, "--socket", "/var/lib/
}
}
So, it invokes /usr/sbin/ maas-dhcp- helper when it expires, triggering an RPC action.
However, when the reserved dynamic range is modified, the dhcpd process is reloaded and the file /var/lib/ maas/dhcp/ dhcpd.leases is cleared. Therefore the actions defined in the file will not be run.
Also, after modifying the reserved dynamic range, I tried running the expiry and release actions manually, but they still do not clear the observed IP. There is something else in MaaS preventing it from being cleared.