netplan ip leases cannot find NetworkManager internal DHCP leases

In 22.04 Desktop aarch64 edition I'm using on an ARM host, the default Netplan renderer is NetworkManager which uses its' own DHCP client in lieu of dhclient. However, Netpan doesn't appear to be respecting this preference for the DHCP client is uses.

Expected Behaviour:
===================

As NetworkManager is the rendered specified in `/etc/netplan/01-network-manager-all.yaml`, it should use NetworkManager's tooling to manage the networking, inclusive of the DHCP client if no alternative was specified.

Diagnostics:
============

Although `sudo systemctl status NetworkManager` states that NetworkManager's own DHCP client is being used, other evidence suggests otherwise.

NetworkManager reports it uses the `internal` DHCP server:
----------------------------------------------------------
sudo systemctl status NetworkManager
● NetworkManager.service - Network Manager
     Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-06-23 12:35:52 BST; 11s ago
       Docs: man:NetworkManager(8)
   Main PID: 6437 (NetworkManager)
      Tasks: 4 (limit: 738)
     Memory: 3.9M
        CPU: 236ms
     CGroup: /system.slice/NetworkManager.service
             └─6437 /usr/sbin/NetworkManager --no-daemon

Jun 23 12:35:52 terrence-desktop systemd[1]: Starting Network Manager...
Jun 23 12:35:52 terrence-desktop systemd[1]: Started Network Manager.
Jun 23 12:35:52 terrence-desktop NetworkManager[6437]: <debug> [1655984152.7199] device[1ef8855a3cc92d44] (lo): hw-addr: unable to read permanent MAC address (use current: 00:00:00:00:>
Jun 23 12:35:52 terrence-desktop NetworkManager[6437]: <info> [1655984152.7210] dhcp-init: Using DHCP client 'internal'
Jun 23 12:35:52 terrence-desktop NetworkManager[6437]: <debug> [1655984152.7260] device[5b1b0e88253999d6] (eth0): speed is now 1000 Mb/s

But `netplan ip leases eth0` thinks otherwise:
----------------------------------------------
sudo nmcli con
NAME UUID TYPE DEVICE
Profile 1 d6b936ad-d73f-4898-a826-edbb61d6155a ethernet eth0
netplan-eth0 626dd384-8b3d-3690-9511-192b2c79b3fd ethernet --

sudo netplan ip leases eth0
No lease found for interface 'eth0': [Errno 2] No such file or directory: '/var/lib/NetworkManager/dhclient-d6b936ad-d73f-4898-a826-edbb61d6155a-eth0.lease'

Remark that the lease file's name is prefixed with "dhclient"; clearly not 'internal' as referenced in the `systemctl status NetworkManager` output above. Also remark that in the `nmcli con` for "netplan-eth0" the specified device is "--"

Attempted Remediation:
======================

For the sake of completeness, I toggled the default value for `managed=` to "true":

    [ifupdown]
    #managed=false
    managed=true

in '/etc/NetworkManager/NetworkManager.conf' and executed

    netplan --debug generate
    netplan --debug apply
    systemctl restart NetworkManager

That did't change the behaviour.

I also attempted to delete the `netplan-eth0` connection so there would be no ambiguity and only one connection available for NetworkManager to use:

    nmcli con delete netplan-eth0

Then I again restarted the Netplan & NetworkManager gears:

        netplan --debug generate
        netplan --debug apply
        systemctl restart NetworkManager

Still no luck.

The output of the 'netplan --debug generate` and `netplan --debug apply` are shown below:

terrence@terrence-desktop:~$ sudo netplan --debug generate
DEBUG:command generate: running ['/lib/netplan/generate']
** (generate:2922): DEBUG: 16:29:46.320: starting new processing pass
** (generate:2922): DEBUG: 16:29:46.320: We have some netdefs, pass them through a final round of validation
** (generate:2922): DEBUG: 16:29:46.320: eth0: setting default backend to 2
** (generate:2922): DEBUG: 16:29:46.320: Configuration is valid
** (generate:2922): DEBUG: 16:29:46.321: Generating output files..
** (generate:2922): DEBUG: 16:29:46.321: networkd: definition eth0 is not for us (backend 2)
** (generate:2922): DEBUG: 16:29:46.322: openvswitch: definition eth0 is not for us (backend 2)
terrence@terrence-desktop:~$ sudo netplan --debug apply
** (generate:2930): DEBUG: 16:29:53.252: starting new processing pass
** (generate:2930): DEBUG: 16:29:53.252: We have some netdefs, pass them through a final round of validation
** (generate:2930): DEBUG: 16:29:53.252: eth0: setting default backend to 2
** (generate:2930): DEBUG: 16:29:53.252: Configuration is valid
** (generate:2930): DEBUG: 16:29:53.254: Generating output files..
** (generate:2930): DEBUG: 16:29:53.254: networkd: definition eth0 is not for us (backend 2)
** (generate:2930): DEBUG: 16:29:53.254: openvswitch: definition eth0 is not for us (backend 2)
DEBUG:no netplan generated networkd configuration exists
DEBUG:netplan generated NM configuration changed, restarting NM
DEBUG:eth0 not found in {}
DEBUG:Merged config:
network:
  ethernets:
    eth0:
      dhcp4: true
  renderer: NetworkManager
  version: 2

DEBUG:Link changes: {}
DEBUG:netplan triggering .link rules for lo
DEBUG:netplan triggering .link rules for eth0
DEBUG:netplan triggering .link rules for wlan0
** (process:2928): DEBUG: 16:29:53.743: starting new processing pass
** (process:2928): DEBUG: 16:29:53.744: We have some netdefs, pass them through a final round of validation
** (process:2928): DEBUG: 16:29:53.744: eth0: setting default backend to 2
** (process:2928): DEBUG: 16:29:53.744: Configuration is valid
DEBUG:eth0 not found in {}
DEBUG:Merged config:
network:
  ethernets:
    eth0:
      dhcp4: true
  renderer: NetworkManager
  version: 2

terrence@terrence-desktop:~$

I increased logging verbosity on eth0 for DHCP in `/etc/NetworkManager/NetworkManager.conf`:

    [logging]
    # https://developer-old.gnome.org/NetworkManager/stable/NetworkManager.conf.html
    #level=INFO
    #level=DEBUG
    domains=ETHER:DEBUG,DHCP4

This revealed other interesting things happening:

`journalctl -u NetworkManager.service | grep DHCP`

    <SNIP>
    Jun 22 14:02:44 terrence-desktop NetworkManager[4833]: <debug> [1655902964.3488] dhcp-init: enabled DHCP client 'dhclient'
    Jun 22 14:02:44 terrence-desktop NetworkManager[4833]: <debug> [1655902964.3489] dhcp-init: enabled DHCP client 'internal'
    Jun 22 14:02:44 terrence-desktop NetworkManager[4833]: <debug> [1655902964.3489] dhcp-init: enabled DHCP client 'systemd' (undocumented internal plugin)
    Jun 22 14:02:44 terrence-desktop NetworkManager[4833]: <debug> [1655902964.3490] dhcp-init:
enabled DHCP client 'nettools' (undocumented internal plugin)

    Jun 22 14:02:44 terrence-desktop NetworkManager[4833]: <info> [1655902964.3491] dhcp-init: Using DHCP client 'internal'
    <SNIP>

Looks like NetworkManager is starting dhclient. Without increasing the verbosity, I'd only have seen the message: "dhcp-init: Using DHCP client 'internal'".

What Worked:
============

Although I could not get NetworkManager's DHCP Client to actually be used- despite what the `status` output said- I DID manage to get 'netplan ip leases eth0` to report the DHCP client info without error by adding the following directive to `NetworkManager.conf`:

    [main]
    dhcp=dhclient

Now the `netplan ip leases <interface>` command succeeds in returning the actually DHCP lease info:

sudo netplan ip leases eth0
[sudo] password for terrence:
lease {
  interface "eth0";
  fixed-address 192.168.1.24;
  option subnet-mask 255.255.255.240;
  option dhcp-lease-time 43200;
  option routers 192.168.1.18;
  option dhcp-message-type 5;
  option dhcp-server-identifier 192.168.1.18;
  option domain-name-servers 192.168.1.18,8.8.8.8;
  option ntp-servers 216.239.35.0;
  option domain-name "F1Linux.com";
  renew 4 2022/06/23 21:12:05;
  rebind 5 2022/06/24 02:27:19;
  expire 5 2022/06/24 03:57:19;
}
lease {
  interface "eth0";
  fixed-address 192.168.1.24;
  option subnet-mask 255.255.255.240;
  option routers 192.168.1.18;
  option dhcp-lease-time 43200;
  option dhcp-message-type 5;
  option domain-name-servers 192.168.1.18,8.8.8.8;
  option dhcp-server-identifier 192.168.1.18;
  option ntp-servers 216.239.35.0;
  option domain-name "F1Linux.com";
  renew 4 2022/06/23 21:38:53;
  rebind 5 2022/06/24 02:33:19;
  expire 5 2022/06/24 04:03:19;
}

The DHCP Client specified in ` systemctl status NetworkManager` now aligns with the the output of `netplan ip leases <interface>`:

sudo systemctl status NetworkManager
● NetworkManager.service - Network Manager
     Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-06-23 18:29:17 BST; 8s ago
       Docs: man:NetworkManager(8)
   Main PID: 2674 (NetworkManager)
      Tasks: 4 (limit: 738)
     Memory: 4.4M
        CPU: 254ms
     CGroup: /system.slice/NetworkManager.service
             └─2674 /usr/sbin/NetworkManager --no-daemon

Jun 23 18:29:17 terrence-desktop systemd[1]: NetworkManager.service: Deactivated successfully.
Jun 23 18:29:17 terrence-desktop systemd[1]: Stopped Network Manager.
Jun 23 18:29:17 terrence-desktop systemd[1]: NetworkManager.service: Consumed 2.177s CPU time.
Jun 23 18:29:17 terrence-desktop systemd[1]: Starting Network Manager...
Jun 23 18:29:17 terrence-desktop systemd[1]: Started Network Manager.
Jun 23 18:29:18 terrence-desktop NetworkManager[2674]: <debug> [1656005358.0158] device[931047a55940e5d6] (lo): hw-addr: unable to read permanent MAC address (use current: 00:00:00:00:>
Jun 23 18:29:18 terrence-desktop NetworkManager[2674]: <info> [1656005358.0161] dhcp-init: Using DHCP client 'dhclient'
Jun 23 18:29:18 terrence-desktop NetworkManager[2674]: <debug> [1656005358.0209] device[b6fce29bc752c44c] (eth0): speed is now 1000 Mb/s

My "Working" Configurations:
============================

I've tried the above even with everything after `dhcp4: yes` commented-out to ensure the netplan stuff was reduced to the simplest config to reduce the chance of a configuration error here breaking things:

`/etc/netplan/01-network-manager-all.yaml`

# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    eth0:
      dhcp4: yes
      dhcp4-overrides:
       use-ntp: true

/etc/NetworkManager/NetworkManager.conf

[main]
plugins=ifupdown,keyfile
dhcp=dhclient

[ifupdown]
#managed=false
managed=true

[device]
wifi.scan-rand-mac-address=no

[logging]
# https://developer-old.gnome.org/NetworkManager/stable/NetworkManager.conf.html
#level=INFO
#level=DEBUG
domains=ETHER:DEBUG,DHCP4

Conclusion:
===========

I've looked at this from a lot of different angles and unless I'm misinterpreting the above- I don't believe so- it appears that Netplan is not respecting the declaration of "NetworkManager" in the `renderer` directive and preferring dhclient instead.