DNS entries for MAAS servers change to secondary IPs

Bug #1999827 reported by Arif Ali
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Committed
Medium
Unassigned
3.1
Fix Committed
Undecided
Unassigned
3.2
Fix Committed
Undecided
Unassigned
3.3
New
Undecided
Unassigned
3.4
Fix Committed
High
Alberto Donato

Bug Description

There is an issue with MAAS that, when you have multiple IP addresses on the same interface, the DNS name for the FQDN of the host can change.

In the scenario were you have MAAS + postgres + HA, you can have 2 IP addresses that are controlled via corosync, and based on where the IP address is, MAAS will automatically create DNS entries, we can see the host resolution[1] and he PTR records [2] as shown. Now the VIPs for maas and postgres are 192.168.210.2 and 192.168.210.3, and hence the FQDN shouldn't really be used for these IP addresses, as ultimately they are secondary.

In a juju deployed environment, in some scenarios we have LMA cross model relations, and the 3 MAAS servers also host telegraf, and then direct their data towards prometheus. This then grabs the IP address of the host as per the relational data (primary interface IP), and grabs the DNS record. In this particular scenario, rather than giving infra1.maas, it will give broam.infra1.maas., which is not the required result.

For a simple reproducer, all you need is a simple MAAS server with single IP address. Then add IP address to the same machine in the same network, and we will see the DNS name of the server fluctuate, below is some output of my reproduced issue on a single MAAS server, noting that each time I am expecting 192.168.1.22 to be my MAAS FQDN

ip a a 192.168.1.8/24 dev eth0

log message in regiond.log

2022-12-15 19:18:05 maasserver.region_controller: [info] Reloaded DNS configuration; ip 192.168.1.8 connected to maas on eth0

~~~
root@maas:~# cat /var/snap/maas/current/bind/zone.1.168.192.in-addr.arpa
<SNIP>
8 30 IN PTR maas.maas.
<SNIP>
22 30 IN PTR eth0.maas.maas.
~~~

ip a a 192.168.1.7/24 dev eth0

~~~
root@maas:~# cat /var/snap/maas/current/bind/zone.1.168.192.in-addr.arpa
<SNIP>
22 30 IN PTR maas.maas.
<SNIP>
7 30 IN PTR eth0.maas.maas.
8 30 IN PTR eth0.maas.maas.
~~~

ip a a 192.168.1.6/24 dev eth0

~~~
root@maas:~# cat /var/snap/maas/current/bind/zone.1.168.192.in-addr.arpa
<SNIP
8 30 IN PTR maas.maas.
<SNIP
6 30 IN PTR eth0.maas.maas.
22 30 IN PTR eth0.maas.maas.
7 30 IN PTR eth0.maas.maas.
~~~

As we can see from the above output, we can see the undesired output.

My expectation is that .22 is always maas.maas. in my scenario, as that should be consistent, no matter what the secondary IP address is set

This has been tested in 2.9.2, as well as my lab with 3.2.6

[1] https://paste.ubuntu.com/p/CZ2wfQgnkx/
[2] https://paste.ubuntu.com/p/7WmYFwb2nT/

### ADDED

As this patch https://bugs.launchpad.net/maas/+bug/1600259, it takes IPs reversely and generate zone file.

https://github.com/maas/maas/blob/0c069f0aafbb650928f6c8536debd0de79eb7484/src/maasserver/dns/zonegenerator.py#L399

e.g

assume there is one nic which has 172.16.116.131 as a primary IP.

after adding one more ip to the same nic, zone file refer to secondary IP instead of primary IP.

{<Domain: name=maas>: defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)}), <Subnet: 172.16.116.0/24:172.16.116.0/24(vid=0)>: defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)}), 'reverse': defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)})}

I printed info from code, xtrusia.maas had 172.16.116.131 in the beginning, but after I added 172.16.116.254, it is changed to later.

Tags: dns sts

Related branches

Revision history for this message
Alberto Donato (ack) wrote :

Does the boot interface for the host change at any time?

Changed in maas:
status: New → Incomplete
Revision history for this message
Seyeong Kim (seyeongkim) wrote :

I've been analyzing this, and I checked that maas picked it as reverse order as

https://bugs.launchpad.net/maas/+bug/1600259

and

https://github.com/maas/maas/blob/master/src/maasserver/dns/zonegenerator.py#L399

Revision history for this message
Seyeong Kim (seyeongkim) wrote :

I can test this in virtual machine easily.

description: updated
Changed in maas:
status: Incomplete → Confirmed
Revision history for this message
Seyeong Kim (seyeongkim) wrote :

Added description but comment as well

As this patch https://bugs.launchpad.net/maas/+bug/1600259, it takes IPs reversely and generate zone file.

https://github.com/maas/maas/blob/0c069f0aafbb650928f6c8536debd0de79eb7484/src/maasserver/dns/zonegenerator.py#L399

e.g

assume there is one nic which has 172.16.116.131 as a primary IP.

after adding one more ip to the same nic, zone file refer to secondary IP instead of primary IP.

{<Domain: name=maas>: defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)}), <Subnet: 172.16.116.0/24:172.16.116.0/24(vid=0)>: defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)}), 'reverse': defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>, {'xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.254'}, 4, None, None), 'ens33.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'172.16.116.131'}, 4, None, 3), 'lxdbr0.xtrusia.maas': HostnameIPMapping('pca68y', 30, {'10.69.50.1'}, 4, None, 3)})}

I printed info from code, xtrusia.maas had 172.16.116.131 in the beginning, but after I added 172.16.116.254, it is changed to later.

Changed in maas:
status: Confirmed → New
Changed in maas:
importance: Undecided → Medium
milestone: none → 3.4.0
status: New → Triaged
tags: added: dns
Revision history for this message
Seyeong Kim (seyeongkim) wrote :

reproduction step further.

0. deploy machine for maas. ( in my case, virtual machine )
1. deploy maas
- sudo snap install maas
- setup maas-test-db
- setup admin

2. ssh into the machine
- ip a
ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    ...
    inet 10.0.0.55/24 brd 10.0.0.255 scope global ens4f0
       valid_lft forever preferred_lft forever
- cat /var/snap/maas/current/bind/zone.0.0.10.in-addr.arpa
...
@ 30 IN NS maas.
55 30 IN PTR node-20.maas.

3. add additional ip to ens4f0
- sudo ip a a 10.0.0.100 dev ens4f0
- ip a
ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:97:25:20 brd ff:ff:ff:ff:ff:ff
    altname enp0s4f0
    inet 10.0.0.55/24 brd 10.0.0.255 scope global ens4f0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global ens4f0
       valid_lft forever preferred_lft forever

4. few mins later
- cat /var/snap/maas/current/bind/zone.0.0.10.in-addr.arpa
...
100 30 IN PTR node-20.maas.
55 30 IN PTR ens4f0.node-20.maas.

Seyeong Kim (seyeongkim)
tags: added: bug-council
Revision history for this message
Jerzy Husakowski (jhusakowski) wrote :

Seyeong, is this reproducible with the latest edge MAAS (3.4.0-beta3)? We've been investigating and fixing a similar issue and the fix recently landed there.

tags: removed: bug-council
Revision history for this message
Seyeong Kim (seyeongkim) wrote :

Hello @jhusakowski

I tested it with 3.4.0-beta3 snap in edge. but it has same behavior

ubuntu@node-20:~$ cat /var/snap/maas/current/bind/zone.0.0.10.in-addr.arpa
; Zone file modified: 2023-06-09 00:07:55.343496.
$TTL 30

...

@ 30 IN NS maas.
51 30 IN PTR node-20.maas.

After

ubuntu@node-20:~$ cat /var/snap/maas/current/bind/zone.0.0.10.in-addr.arpa
; Zone file modified: 2023-06-09 00:10:09.763022.
$TTL 30

...

@ 30 IN NS maas.
100 30 IN PTR node-20.maas.
51 30 IN PTR ens4f0.node-20.maas.

tags: added: bug-council
Revision history for this message
Seyeong Kim (seyeongkim) wrote :

I analyzed a bit more, reverse is not the thing.

actually mappings Subnet has already modified values.

keep researching..

{
  <Domain: name=maas>:
    defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>,
    {
      'node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.100'}, 4, None, None),
      'ens4f0.node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.51'}, 4, None, 1)
    }),
  <Subnet: 10.0.0.0/24:10.0.0.0/24(vid=0)>:
    defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>,
    {
      'node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.100'}, 4, None, None),
      'ens4f0.node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.51'}, 4, None, 1)
    }
  ),
  'reverse':
    defaultdict(<class 'maasserver.models.staticipaddress.HostnameIPMapping'>,
    {
      'node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.100'}, 4, None, None),
      'ens4f0.node-20.maas': HostnameIPMapping('hre8hh', 30, {'10.0.0.51'}, 4, None, 1)
    }
  )
}

Revision history for this message
Seyeong Kim (seyeongkim) wrote :

I checked code a bit more.

https://pastebin.ubuntu.com/p/jYhdxqKQfR/

one more left outer join make order changed

i think this is the point.

Could you please give me any advices?

Thanks.

Revision history for this message
Seyeong Kim (seyeongkim) wrote :

I proposed merge request

Revision history for this message
Jerzy Husakowski (jhusakowski) wrote :

There's a newer fix in 3.4.0, it'd be interesting to know if you can reproduce the issue using that fix. Could you test it on MAAS from this ppa? https://launchpad.net/~cgrabowski/+archive/ubuntu/maas-next

tags: removed: bug-council
Revision history for this message
Seyeong Kim (seyeongkim) wrote :

@jhusakowski

I can't install that ppa I see build is failed.

I'll test after fixing it.

Thanks in advance.!

Alberto Donato (ack)
Changed in maas:
milestone: 3.4.0 → 3.4.x
Revision history for this message
Arif Ali (arif-ali) wrote :

@Jerzy, I am running the 3.4/edge snap

root@maas:~# snap list maas
Name Version Rev Tracking Publisher Notes
maas 3.4.0~rc1-14285-g.358798bbf 29143 3.4/edge canonical✓ -

I am presuming this should have the aforementioned fix, if so, I was able to reproduce this in my lab

root@maas:~# cat /var/snap/maas/current/bind/zone.1.168.192.in-addr.arpa | grep maas.maas
6 30 IN PTR maas.maas.
22 30 IN PTR eth0.maas.maas.

root@maas:~# grep -E "192.168.1.6|192.168.1.22" /var/snap/maas/current/bind/*
grep: /var/snap/maas/current/bind/cache: Is a directory
/var/snap/maas/current/bind/zone.maas:maas 30 IN A 192.168.1.6
/var/snap/maas/current/bind/zone.maas:eth0.maas 30 IN A 192.168.1.22
/var/snap/maas/current/bind/zone.maas-internal:192-168-1-0--24 15 IN A 192.168.1.6
/var/snap/maas/current/bind/zone.maas-internal:192-168-1-0--24 15 IN A 192.168.1.22

Changed in maas:
status: Triaged → Fix Committed
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

MAAS bug-council,

Can this fix be included/targeted to series 3.3 and 3.2 as well?

If so, SEG will proceed with older series afterward.

Thanks,
Mauricio

tags: added: bug-council
Revision history for this message
Jerzy Husakowski (jhusakowski) wrote :

The change proposed by Seyeong landed in master (3.5), but not yet in 3.4. We targeted the issue to 3.4 as well and need to confirm the fix. The intention is to backport the fix to all affected supported revisions.

Changed in maas:
milestone: 3.4.x → 3.5.0
tags: removed: bug-council
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.