MAAS

MAAS 3.4 RC (Aug 2nd 2023) breaks DNS

Bug #2029481 reported by Kevin Reeuwijk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Status tracked in 3.5
3.3
Fix Committed
Critical
Christian Grabowski
MAAS 3.3.x
3.4
Fix Released
Critical
Christian Grabowski
MAAS 3.4.0-rc1
3.5
Fix Committed
Critical
Christian Grabowski
MAAS 3.5.0

Bug Description

After the MAAS snap in the 3.4/beta channel auto-upgraded to the 3.4.0~rc1-14297-g.b27476324 build, DNS no longer functions properly. New entries in the DNS zones in MAAS do not show up in their respective zone files and DNS queries for these new records fail.

Restarting or reloading the MAAS snap will populate the zone files and get queries for the new records to resolve, but this action causes in-flight PXE deployments to crash as all of MAAS restarts. This solution also isn't practical for us since we deploy Kubernetes clusters with MAAS and need DNS to work immediately, otherwise the cluster fails to build.

As I suspected that maybe this had something to do with AppArmor, I tried disabling it and setting the profiles to complain mode, but none of these had any positive effect. So this was rolled back to the original AppArmor configuration.

Regiond logs are attached.

Related branches

~cgrabowski/maas:backport_fix_dns_db_access_out_of_thread_to_3.3
MAAS Lander: Approve
Christian Grabowski: Approve
Diff: 47 lines (+6/-5)
2 files modified
src/maasserver/region_controller.py (+4/-1)
src/maasserver/tests/test_region_controller.py (+2/-4)
~cgrabowski/maas:backport_fix_dns_db_access_out_of_thread_to_3.4
Christian Grabowski: Approve
Diff: 47 lines (+6/-5)
2 files modified
src/maasserver/region_controller.py (+4/-1)
src/maasserver/tests/test_region_controller.py (+2/-4)
~cgrabowski/maas:fix_dns_db_access_out_of_thread
MAAS Lander: Approve
Alberto Donato: Approve
Diff: 47 lines (+6/-5)
2 files modified
src/maasserver/region_controller.py (+4/-1)
src/maasserver/tests/test_region_controller.py (+2/-4)
Revision history for this message
Kevin Reeuwijk (kreeuwijk) wrote :
Revision history for this message
Christian Grabowski (cgrabowski) wrote :

seems there's a DB query outside of a database thread.

Changed in maas:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Christian Grabowski (cgrabowski)
milestone: none → 3.5.0
Revision history for this message
Anton Troyanov (troyanov) wrote :

Hi Kevin,

Thanks for the bug report. We are planning to land a fix for 3.4 asap but it will take some time for us to make it to 3.4/beta channel.

Meanwhile maybe you can try 3.4/edge (after a new version is published) to check if that actually solves the issue?

Revision history for this message
Kevin Reeuwijk (kreeuwijk) wrote :

Hi Anton,

yes I can easily try a 3.4/edge build when you've made it available. Just ping me and I'll get right on it.

Revision history for this message
Anton Troyanov (troyanov) wrote :

Kevin, 3.4.0~rc1-14298-g.8b414218b is now available in 3.4/edge

Revision history for this message
Kevin Reeuwijk (kreeuwijk) wrote :

Upgrade to 3.4.0~rc1-14298 succeeded, testing now

Revision history for this message
Kevin Reeuwijk (kreeuwijk) wrote (last edit ):

Anton,

I can confirm the edge release works, deployments are working again. Thanks!
I have one lingering issue, as you can see from the attached screenshot.

Deleting a DNS record causes a ghost record to appear in both the MAAS database as well as the actual zonefile. The ghost record has the form of A-B-C-D for an IP address of a.b.c.d.
It is impossible to delete the record from the GUI.

This happens with both 3.4.0~rc1-14297-g.b27476324 and the new 3.4.0~rc1-14298-g.8b414218b version.
While not a huge problem, it would be appreciated if this can be fixed in the next drop.

-Kevin

Revision history for this message
Anton Troyanov (troyanov) wrote :

Kevin,

This happens because after DNS record is removed, MAAS still tracks IP address marked as USER_RESERVED.
You can find it in the database table `maasserver_staticipaddress`

I agree that it is not very clear from the UX perspective, but as a workaround I can suggest you to use MAAS CLI to remove unwanted record: `maas $LOGIN ipaddresses release ip=a.b.c.d`

I've checked 3.2 and 3.3 and it behaves exactly the same way.

But if you consider this to be a bug, may I ask you to open a new bug in LP, so we can track it separately.

Thanks in advance!

Revision history for this message
Kevin Reeuwijk (kreeuwijk) wrote :

Hi Anton,

thank you for the workaround to clean them up, I will try that.

It also happens when I manually create a DNS record (for a random IP) in the MAAS GUI and then delete it. So I don't think this is intended to work that way, it didn't in prior releases at least. So I'll raise a bug for it.

-Kevin

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.