Ubuntu 22.04.2, nsupdate stopped recognizing HMAC-MD5 key after update from 1:9.18.1-1ubuntu1.3 to 1:9.18.12-0ubuntu0.22.04.1

Bug #2015176 reported by Wladimir Mutel
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
BIND
New
Undecided
Unassigned
bind9 (Ubuntu)
Fix Released
Undecided
Lena Voytek
Jammy
Fix Committed
Undecided
Lena Voytek
Kinetic
Won't Fix
Undecided
Lena Voytek
Lunar
Fix Committed
Undecided
Lena Voytek

Bug Description

[Impact]

Bind9 upstream accidentally introduced a regression that made old HMAC-MD5 key pair files unreadable in version 9.18.8.

This capability was fixed with the release of 9.18.17 through https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/8069. This means the issue will be fixed alongside the MRE release of 9.18.18 in Lunar and Jammy.

The issue is fixed by restoring the missing files and pointing to them correctly.

[Test Plan]

To test that this fix specifically is successful, you can run:

# lxc launch ubuntu:{lunar, jammy} test-bind9
# lxc exec test-bind9 bash

# apt update && apt dist-upgrade -y
# apt install bind9

Create example key files since HMAC-MD5 is deprecated and creation of them was removed from focal onward
# cat <<EOF >Kexample.com.+157+15178.key
example.com. IN KEY 512 3 157 SItPKKvb7T9QEBRl9Mmrng==
EOF

# cat <<EOF >Kexample.com.+157+15178.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: SItPKKvb7T9QEBRl9Mmrng==
Bits: AAA=
Created: 20230920212628
Publish: 20230920212628
Activate: 20230920212628
EOF

# nsupdate -k Kexample.com.+157+15178.private

Prior to the fix, this results in something like:
20-Sep-2023 21:41:40.730 Kexample.com.+157+15178.private:1: unknown option 'Private-key-format:'
20-Sep-2023 21:41:40.730 Kexample.com.+157+15178.private:8: unexpected token near end of file
could not read key from Kexample.com.+157+15178.{private,key}: unexpected token

After the fix, the command should succeed with a possible deprecation warning:
20-Sep-2023 21:36:24.723 Kexample.com.+157+15178.private: Use of K* file pairs for HMAC is deprecated

[Where problems could occur]

Problems with this release would most likely occour outside the scope of this issue, as the MRE release includes many other fixes and updates alongside this. However, issues related directly to this change would likely revolve around other key files breaking or the HMAC-MD5 files not being restored properly to match their original state.

[Original Description]

I have a key with the following contents (key material replaced with ...) :

+ cat /etc/bind/Khost.+157+35878.key
host. IN KEY 0 3 157 YSp... ...QsQ==

+ cat /etc/bind/Khost.+157+35878.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: YSp......QsQ==
Bits: AAA=
Created: 20180616045813
Publish: 20180616045813
Activate: 20180616045813

it worked for long time, up till package version 1:9.18.1-1ubuntu1.3
but since upgrading to 1:9.18.12-0ubuntu0.22.04.1 , it stopped working with nsupdate giving out the following :

Creating key...
could not read key from /etc/bind/Khost.+157+35878.{private,key}: file not found

in strace printout, I see that nsupdate successfully opens and reads .private key file but then checks existence of the same file name without suffix (as specified after nsupdate -k) and fails.

were there any changes in key parsing from 9.18.1 to 9.18.13 ?
reverting bind9-utils, bind9-dnsutils and bind9-libs back to 1:9.18.1-1ubuntu1.3 restored the desired behavior.
please advise if I should fix the key format after the upgrade, or if this is a regression to be fixed from your side.

Related branches

Wladimir Mutel (mwg)
description: updated
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thank you for taking the time to report this bug.

As you have noticed, bind9 has been updated to its latest version in the 9.18.x series. This was quite a leap, which brought many important bugfixes to Ubuntu users across many release. We do our best to guarantee that these bigger-than-normal updates go smooth, but unfortunately there are scenarios which are too complex or uncommon to cover. It seems that you've hit one of those bumps.

I was able to successfully reproduce the problem you reported after I generated an HMAC-MD5 key on Bionic (which is the only LTS distro that's still able to generate keys using this algorithm), by doing:

$ dnssec-keygen -a HMAC-MD5 -b 128 -n HOST bind9.lxd
$ cat Kbind9.lxd.+157+25064.key
bind9.lxd. IN KEY 512 3 157 0OaUhv7uRii9yrq6kdSnbA==
$ cat Kbind9.lxd.+157+25064.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: 0OaUhv7uRii9yrq6kdSnbA==
Bits: AAA=
Created: 20230405201738
Publish: 20230405201738
Activate: 20230405201738

This effectively created a pair of keys, which I transferred to a Jammy container and then tried invoking "nsupdate -k":

# nsupdate -k /etc/bind/Kbind9.lxd.+157+25064
could not read key from /etc/bind/Kbind9.lxd.+157+25064.{private,key}: file not found

I spent some time debugging the problem, and was able to track down the issue to the fact that bind9 has chosen different numbers to represent key algorithms internally. These numbers happen to be hardcoded inside the keys, which means that, when nsupdate checks if the specified algorithm is supported (and HMAC-MD5 is still supported), it can't find the entry associated with the old number and bails out.

In order to make things work again, a quick workaround is to edit your key files and replace "157" with "160". For example, here are the two files listed above with the modifications in place:

# cat Kbind9.lxd.+157+25064.key
bind9.lxd. IN KEY 512 3 160 0OaUhv7uRii9yrq6kdSnbA==
# cat Kbind9.lxd.+157+25064.private
Private-key-format: v1.3
Algorithm: 160 (HMAC_MD5)
Key: 0OaUhv7uRii9yrq6kdSnbA==
Bits: AAA=
Created: 20230405201738
Publish: 20230405201738
Activate: 20230405201738

There is also a newer format for key files which you can use. For the example I'm using here, it looks like this:

# cat Kbind9.lxd.+157+25064
key "bind9.lxd" {
        algorithm hmac-md5;
        secret "0OaUhv7uRii9yrq6kdSnbA==";
};

Note that this format doesn't contain any internal references to how bind9 numbers the algorithms, which is more future-proof. An even better solution would be to move away from HMAC-MD5, but I understand that that's not always possible.

For reference, the upstream commit that broke things was:

https://gitlab.isc.org/isc-projects/bind9/-/commit/09f7e0607a34d90eae53f862954e98c31b5ae532

There's an upstream bug about this problem, which I will link in a moment.

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :
Changed in bind9 (Ubuntu):
status: New → Triaged
Changed in bind9 (Ubuntu Jammy):
status: New → Triaged
Changed in bind9 (Ubuntu Kinetic):
status: New → Triaged
tags: added: regression-update
Revision history for this message
Robie Basak (racb) wrote :

I wonder if there are any plans to fix this, or are we going to have to leave it now that the damage is done?

tags: added: server-triage-discuss
tags: added: server-todo
removed: server-triage-discuss
Revision history for this message
Lena Voytek (lvoytek) wrote :

For now I would like to avoid a second regression for users of Ubuntu stable releases, so I will avoid updates unless there is enough demand for them. If anyone else has experienced this issue too and comes across this, please let me know. As for mantic, I'll wait and see if upstream comes up with a solution in 9.19.x and port it in accordingly for future consistency. Thanks!

Changed in bind9 (Ubuntu):
assignee: nobody → Lena Voytek (lvoytek)
Changed in bind9 (Ubuntu Jammy):
assignee: nobody → Lena Voytek (lvoytek)
Changed in bind9 (Ubuntu Kinetic):
assignee: nobody → Lena Voytek (lvoytek)
Changed in bind9 (Ubuntu Lunar):
assignee: nobody → Lena Voytek (lvoytek)
Revision history for this message
Lena Voytek (lvoytek) wrote :

Still waiting on upstream to make a decision, removing server-todo for now

tags: removed: server-todo
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package bind9 - 1:9.18.18-0ubuntu1

---------------
bind9 (1:9.18.18-0ubuntu1) mantic; urgency=medium

  * New upstream release 9.18.18 (LP: #2034367)
    - Updates:
      + Mark a primary server as temporarily unreachable when a TCP connection
        response to an SOA query times out, matching behavior of a refused TCP
        connection.
      + Mark dialup and heartbeat-interval options as deprecated.
      + Retry DNS queries without an EDNS COOKIE when the first response is
        FORMERR with the EDNS COOKIE that was sent originally.
      + Use NS records for the relaxed QNAME minimization mode to reduce the
        number of queries from named.
    - Bug Fixes:
      + Fix assertion failure from processing already-queued queries while
        server is being reconfigured or cache is being flushed.
      + Fix failure to load zones containing resource records with a TTL value
        larger than 86400 seconds when dnssec-policy is set to insecure.
      + Fix the ability to read HMAC-MD5 key files (LP: #2015176).
      + Fix stability issues with the catalog zone implementation.
    - See https://bind9.readthedocs.io/en/v9.18.18/notes.html for additional
      information.

 -- Lena Voytek <email address hidden> Tue, 05 Sep 2023 13:20:06 -0700

Changed in bind9 (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Wladimir Mutel (mwg) wrote :

great that it is fixed for mantic, how about fixing it for jammy as well ?

Lena Voytek (lvoytek)
Changed in bind9 (Ubuntu Kinetic):
status: Triaged → Won't Fix
Changed in bind9 (Ubuntu Lunar):
status: Triaged → In Progress
Changed in bind9 (Ubuntu Jammy):
status: Triaged → In Progress
Lena Voytek (lvoytek)
description: updated
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Wladimir, or anyone else affected,

Accepted bind9 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/bind9/1:9.18.18-0ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in bind9 (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Changed in bind9 (Ubuntu Lunar):
status: In Progress → Fix Committed
tags: added: verification-needed-lunar
Revision history for this message
Steve Langasek (vorlon) wrote :

Hello Wladimir, or anyone else affected,

Accepted bind9 into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/bind9/1:9.18.18-0ubuntu0.23.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Lena Voytek (lvoytek) wrote :

Verified for Lunar and Jammy

# lxc launch ubuntu:jammy test-bind9-jammy
# lxc exec test-bind9-jammy bash

cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

# apt update && apt dist-upgrade -y
# apt install bind9 -y

# cat <<EOF >Kexample.com.+157+15178.key
example.com. IN KEY 512 3 157 SItPKKvb7T9QEBRl9Mmrng==
EOF

# cat <<EOF >Kexample.com.+157+15178.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: SItPKKvb7T9QEBRl9Mmrng==
Bits: AAA=
Created: 20230920212628
Publish: 20230920212628
Activate: 20230920212628
EOF

# nsupdate -k Kexample.com.+157+15178.private

29-Sep-2023 22:23:44.897 Kexample.com.+157+15178.private: Use of K* file pairs for HMAC is deprecated

# lxc launch ubuntu:lunar test-bind9-lunar
# lxc exec test-bind9-lunar bash

cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

# apt update && apt dist-upgrade -y
# apt install bind9 -y

# cat <<EOF >Kexample.com.+157+15178.key
example.com. IN KEY 512 3 157 SItPKKvb7T9QEBRl9Mmrng==
EOF

# cat <<EOF >Kexample.com.+157+15178.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: SItPKKvb7T9QEBRl9Mmrng==
Bits: AAA=
Created: 20230920212628
Publish: 20230920212628
Activate: 20230920212628
EOF

# nsupdate -k Kexample.com.+157+15178.private

29-Sep-2023 22:51:23.253 Kexample.com.+157+15178.private: Use of K* file pairs for HMAC is deprecated

tags: added: verification-done verification-done-jammy verification-done-lunar
removed: verification-needed verification-needed-jammy verification-needed-lunar
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.