freeipa server install fails - named-pkcs11 fails to run

See also https://bugs.launchpad.net/ubuntu/+source/freeipa/+bug/1765616/comments/9

must be a race condition again, I can't reproduce it here

I'm doing this in a LXC container. Could that be of influence?

doesn't hurt to try on qemu/kvm or actual hw

My hostname was not a FQDN. After I changed it to be FQDN, and made sure the entry
is in /etc/hosts, the installation continues.

However, there is still a problem. The nameserver fails to (re)start.

Configuring DNS (named)
  [1/11]: generating rndc key file
  [2/11]: adding DNS container
  [3/11]: setting up our zone
  [4/11]: setting up our own record
  [5/11]: setting up records for other masters
  [6/11]: adding NS record to the zones
  [7/11]: setting up kerberos principal
  [8/11]: setting up named.conf
  [9/11]: setting up server configuration
  [10/11]: configuring named to start on boot
  [11/11]: changing resolv.conf to point to ourselves
Done configuring DNS (named).
Restarting the web server to pick up resolv.conf changes
Configuring DNS key synchronization service (ipa-dnskeysyncd)
  [1/7]: checking status
  [2/7]: setting up bind-dyndb-ldap working directory
  [3/7]: setting up kerberos principal
  [4/7]: setting up SoftHSM
  [5/7]: adding DNSSEC containers
  [6/7]: creating replica keys
  [7/7]: configuring ipa-dnskeysyncd to start on boot
Done configuring DNS key synchronization service (ipa-dnskeysyncd).
Restarting ipa-dnskeysyncd
Restarting named
Updating DNS system records
ipapython.dnsutil: ERROR DNS query for usrv1.ijtest.nl. 1 failed: The DNS operation timed out after 30.000865221 seconds
ipaserver.dns_data_management: ERROR unable to resolve host name usrv1.ijtest.nl. to IP address, ipa-ca DNS record will be incomplete

In syslog there is this:

May 6 20:18:01 usrv1 named-pkcs11[25219]: ../../../lib/dns-pkcs11/view.c:962: REQUIRE(view->zonetable != ((void *)0)) failed, back trace
May 6 20:18:01 usrv1 named-pkcs11[25219]: #0 0x55ceb0cb4cc0 in ??
May 6 20:18:01 usrv1 named-pkcs11[25219]: #1 0x7f4ae89007fa in ??
May 6 20:18:01 usrv1 named-pkcs11[25219]: #2 0x7f4ae93122aa in ??
May 6 20:18:01 usrv1 named-pkcs11[25219]: #3 0x55ceb0cd2a77 in ??
May 6 20:18:01 usrv1 named-pkcs11[25219]: #4 0x55ceb0c967d1 in ??
May 6 20:18:01 usrv1 named-pkcs11[25219]: #5 0x55ceb0cdf309 in ??
May 6 20:18:01 usrv1 named-pkcs11[25219]: #6 0x55ceb0ce0f33 in ??
May 6 20:18:01 usrv1 named-pkcs11[25219]: #7 0x7f4ae8927b59 in ??
May 6 20:18:01 usrv1 named-pkcs11[25219]: #8 0x7f4ae7ea16db in ??
May 6 20:18:01 usrv1 named-pkcs11[25219]: #9 0x7f4ae75d588f in ??
May 6 20:18:01 usrv1 named-pkcs11[25219]: exiting (due to assertion failure)

yep, that's a known issue, though it doesn't have a bug for it so maybe this should be it

the installation shouldn't start if the hostname is not a FQDN though, so that's another bug then

Do you want me to create a bugreport for that non-FQDN?

When you said: "yep, that's a known issue" you referred to the non-FQDN. But the above
error is after I corrected that. So, with a FQDN.

BTW, I'm doing the install with --setup-dns. Is that what you do as well?
At the end of the installation the nameserver (bind9-pkcs11) does not start anymore.

I mean the dns setup is known to be broken, I don't know why it gets an empty zone from ldap and reported it upstream but the next step would be to debug with gdb and I didn't get anywhere with it yet..

Hi guys, I'm getting the same while installing on real hardware. The name server refuses to start up with the following error in the logs:

../../../lib/dns-pkcs11/view.c:962: REQUIRE(view->zonetable != ((void *)0)) failed, back trace

Using the server's FQDN.

Installing on Ubuntu 18.04 using ipa-server-install --setup-dns. Here's the package version info:
freeipa-server | 4.7.0~pre1+git20180411-2ubuntu2 | http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
bind9 | 1:9.11.3+dfsg-1ubuntu1 | http://us.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
bind9-dyndb-ldap | 11.1-3ubuntu1 | http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

@Timo what is the named command that you used to debug? I can't get named
to produce the same error (at view.c:962) when I run it as follows (this
is the command I found in the log):

/usr/sbin/named-pkcs11 -f -u bind

or

/usr/sbin/named-pkcs11 -g -u bind

It crashes at:
08-May-2018 07:07:41.154 ../../../lib/isc-pkcs11/md5.c:93: fatal error:
08-May-2018 07:07:41.154 RUNTIME_CHECK(pk11_get_session(ctx, OP_DIGEST, isc_boolean_true, isc_boolean_false, isc_boolean_false, ((void *)0), 0) == 0) failed

you need to prime it with the environment:

SOFTHSM2_CONF=/etc/ipa/dnssec/softhsm2.conf KRB5_KTNAME=/etc/bind/named.keytab gdb --args named-pkcs11 -g -u bind

then the problem is that there are no debug symbols for named-pkcs11, not even in bind9-dbgsym and I've no idea why..

I have debug symbols, I installed bind9-dbgsym libisc169-dbgsym, but you
probably did that as well, right?

Reading symbols from /usr/sbin/named-pkcs11...Reading symbols from /usr/lib/debug/.build-id/a6/b02914ac626d6db7786c640335d7e674d21dcc.debug...done.

Not that it helped me any further without having looked at the named
source code.

No symbol info for the library :-(

Installing libdns-export1100-dbgsym libdns1100-dbgsym libisc-export169-dbgsym
helped. I now have debug symbols in view.c

lucky you

Reading symbols from /usr/sbin/named-pkcs11...(no debugging symbols found)...done.

I have all the dbgsym packages installed..

Status changed to 'Confirmed' because the bug affects multiple users.

Any news on this bug?

I discovered that if I replace /usr/sbin/named-pkcs11 with the /usr/sbin/named executable, everything seems to work fine. However, I do not know what are be the possible consequences of this change.

interesting, maybe there's something wrong with bind9 build..

Maybe. Note that if you try to execute named directly (instead of named-pkcs11), it will fail since the AppArmor profile for named forbid the loading of the ldap plugin.

only if you put it in enforce mode, it's in complain mode by default

For some reason, I have /usr/sbin/named in enforce mode by default (I am sure I did not change anything manually). Ubuntu 18.04 installed with an alternate CD on a KVM virtual machine.

Is there a recommended workaround? For example, install without DNS support and use a separate bind installation?

I think the my trick (copy /usr/sbin/named into /usr/sbin/named-pkcs11) works quite well. Not sure about the differences between named and named-pkcs11, but I think it is essentially the fact that named-pkcs11 supports cryptographic devices while plain named doesn't. In order to avoid /usr/sbin/named-pkcs11 to be rewritten during an update, you may want to use dpkg-divert.

no, bind9 needs to be fixed instead, the way it's build got revamped in 9.11.3+dfsg-1 and I believe that's what broke it..

I'll take a look

I can ask Ondrej too

Thanks, he definitely knows more about bind than I do :)

Note that named-pkcs11 only crashes at startup when the section

dyndb "ipa" "/usr/lib/bind/ldap.so"

is present. If commented out, the daemon starts (although it becomes useless in this context).

I tried the new version of bind (1:9.11.3+dfsg-1ubuntu1.1) but the -pkcs11 version still crashes with the ldap plugin.

Has anybody reproduced this on debian? I confirm it happening then deploying freeipa, but I'm also looking at a simpler test case now.

Outside of freeipa, I can get it to crash with multiple different assertion errors, just not yet the one we get when using it with freeipa. It doesn't look like a very robust system.

This is a recipe with all the work-arounds needed to get a freeipa server with integrated DNS going on Ubuntu bionic/18.04 LTS or later.

Without these workarounds, you will hit so many bugs the system is uninstallable as of 6/23/18.

I chose Lubuntu as a platform as I wanted an integrated browser as a way to check for good operations without the complexity of most of the networking stack in the chain.

I started with a ‘clean install’ of Lubuntu 18.04. I needed 4 CPU cores and 4GB of memory to avoid most of the race conditions that kill the installer. You can reduce these to 1 core and 2GB after installation (that’s a ‘low demand minimum’).

Where you see 192.168.50.64 below, replace that with the IP address of your freeipa machine.
Where you see ri.mamabosso.com below, replace that with the private IP address range of the sub-domain you’ll use for the freeipa server. (If your public domain is xyz.com, it’s best practice to add a subdomain for the private addresses, so local.xyz.com to resolve them. Split-view and the like generate more problems than they solve).

You should see no error messages at any point in this process. If you do, stop to puzzle them out before moving on.

Get to a command prompt as root:

apt update
apt upgrade

apt install freeipa-server-dns python-psutil haveged

Cause /etc/hosts to look like:

127.0.0.1 localhost
192.168.50.64 directory1.ri.mamabosso.com directory1
127.0.1.1 directory1.ri.mamabosso.com directory1

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Cause /etc/hostname to look like:
directory1.ri.mamabosso.com

Remove anything in /etc/netplan unless your sure otherwise. In /etc/netplan add file:
/etc/netplan/01-networkd.yaml with the below (change addresses and domains to yours):

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.50.64/24
      gateway4: 192.168.50.1
      nameservers:
          search: [ri.mamabosso.com, mamabosso.com]
          addresses: [127.0.0.1]

These commands are needed to avoid several bugs later on:

systemctl disable systemd-resolved
systemctl disable network-manager
systemctl disable NetworkManager
mv /lib/systemd/system/NetworkManager.service NetworkManager.service.res
usermod bind -aG softhsm
mkdir /var/lib/softhsm/tokens
chown root:softhsm /var/lib/softhsm/tokens
chmod 0770 /var/lib/softhsm/tokens
chmod g+s /var/lib/softhsm/tokens
mv /usr/sbin/named-pkcs11 /usr/sbin/named-pkcs11-dpkg-dist
cp /usr/sbin/named /usr/sbin/named-pkcs11
#The dependency on named-pkcs11 is a fedora legacy and is no longer necessary
#which is fortunate as named-pkcs11 crashes on startup leaving the system with
#no resolver.

Make /etc/resolv.conf:

nameserver 127.0.0.1
search <your local domain here, ri.mamabosso.com in my case>

patch freeipa’s installer to avoid race conditions that otherwise would crash it:

Note: you should exactly match the indenting you find in the programs to be edited below, using spaces and not tabs.

in /usr/lib/python2.7/dist-packages/ipaserver/install/cainstance.py after
...

P.S. After the systemctl disable commands, you may need to delete the '/etc/resolv.conf' then make a new one with the simple content as it could be a link to a stub for systemd-resolved.

PPS. Freeipa needs fontawesome version 4 or you get unicode boxes. Bionic ships v3. Attached find v4. put them in /usr/share/fonts/fontawesome

PPPS. You don't need the latest fontsawesome after all for the gui to work. However, you do need:

apt install libjs-scriptaculous

and

The installed code expects fontawesome, not font-awesome in the truetype directory.

cd /usr/share/fonts/truetype
ln -s font-awesome /usr/share/fonts/truetype/fontawesome

(the debian/ubuntu world installs with a -, the rhel/centos/fedora world as fontawesome.

It's not useful to stuff every workaround on an unrelated bug, you should've searched the existing ones first and file new bugs when necessary:

fontawesome path: https://bugs.launchpad.net/ubuntu/+source/freeipa/+bug/1772921
/var/lib/krb5kdc permissions https://bugs.launchpad.net/ubuntu/+source/freeipa/+bug/1772447
systemd-resolved breaking things https://bugs.launchpad.net/ubuntu/+source/freeipa/+bug/1772405

looking at your list only the missing libjs-scriptaculous dependency is a new issue

and you can see from that bind-dyndb-ldap commit it's only about the rpm dropping the dependency, we never had that anyway, since bind-pkcs11 is not a separate package

I've installed FreeIPA server with all the patches mentioned here but "sudo ipactl status" shows that kadmin services is stopped. I had to create an empty file /etc/krb5kdc/kadm5.acl which looks like solved the problem. Not sure if it is the right approach.

Another issue I have is that when in Web UI I click on Authentication is complains about "IPA Error 4301: CertificateOperationError" "Certificate operation cannot be completed: Unable to communicate with CMS (Start tag expected, '<' not found, line 1, column 1)"

Both bugs have been already reported: see https://bugs.launchpad.net/ubuntu/+source/freeipa/+bug/1772205 and https://bugs.launchpad.net/ubuntu/+source/freeipa/+bug/1772450. The freeipa/staging ppa (https://launchpad.net/~freeipa/+archive/ubuntu/staging) contains a new version of freeipa which fixes many bugs which have been reported (including these).

Some error documented here:
https://bugzilla.redhat.com/show_bug.cgi?id=1410433

Here's the referenced commit for the fix for fedora's bind9 code:
https://pagure.io/fedora-bind/c/3d5ea105bd877f0069452e450320f8877b01cb52?branch=master

good catch.. I did have a look at the fedora patches there but didn't go deep enough.. that's an old patch which never went upstream :/

uploaded bind to the staging ppa, please test once it's built

Looks like bind9 is fixed! Install completes with no issues and named-pks11 runs without crashing.

> Looks like bind9 is fixed! Install completes with no issues and named-pks11 runs without crashing.

Great! Thank you for the report.

I'm not sure this bug was ever clear on exactly what the problem was with bind9, in terms of bind9. And if it is now fixed, I don't know when it was fixed. So I'll mark the bind9 task Incomplete for now. If someone wants to describe the bind9 bug in terms of bind9 itself, and/or describe when it was fixed, we can update the status appropriately.

I think he means it was fixed with the package that Timo upload to the ppa, not with the package in the archive.

@ahasenack is correct.

@racb, this bug is fixed in the sense that I found the appropriate patches missing from bind9, and the staging version that @tjaalton built and uploaded stops the crashes.

This is the patch applied
https://pagure.io/fedora-bind/c/3d5ea105bd877f0069452e450320f8877b01cb52?branch=master

Any ETA on when/if this will be fixed. I am trying to add a new freeipa server running 18.04 by adding it as a replica to a current setup, but it seems to fail on Bind and what is described here.

I tried the PPA listed above but when adding as a replica using those packages Igot this error if anyone here is interested.

error] UNWILLING_TO_PERFORM: {'info': u'modification of attribute nsds5ReplicaReleaseTimeout is not allowed in replica entry', 'desc': u'Server is unwilling to perform'}

@garyx that is unrelated. Open a new bug and please fully post the debug output and logs, rather than an out-of-context snippit.

@Gabriel thanks, I follow now.

@Timo do you have plans on getting this landed please? Or do you want the server team to do it?

We can take on this.

the patch is on debian git, but hasn't been uploaded there yet it seems

and yes the server team is free to handle this, but I can step in too if it helps

I just finished an ipa-server-install run on cosmic where I hit the abort error. But when using the patched bind9 package from https://launchpad.net/~kstenerud/+archive/ubuntu/bind9-rtld-deepbind-1769440/ which has the patch from fedora and Timo, it worked.

Timo, where is the patch in debian git? I'm looking at <email address hidden>:dns-team/bind9.git (https://salsa.debian.org/dns-team/bind9) but can't find it. It's currently at debian/1%9.11.4+dfsg-4 which is what was released last.

I also checked https://salsa.debian.org/dns-team/bind

I wonder if this patch shouldn't be applied only when doing the -pkcs11 rebuild

I don't think there is anything to do here for freeipa itself, just bind9. Marking the freeipa task as invalid.

bah, looks like I didn't push it back then.. is pushed now

This bug was fixed in the package bind9 - 1:9.11.4+dfsg-3ubuntu2

---------------
bind9 (1:9.11.4+dfsg-3ubuntu2) cosmic; urgency=medium

  * d/p/skip-rtld-deepbind-for-dyndb.diff: Add a patch to fix named-pkcs11
    crashing on startup. (LP: #1769440)

 -- Karl Stenerud <email address hidden> Thu, 30 Aug 2018 07:11:39 -0700

Will this be fixed in bionic where IPA is currently broken and unusable?

Can we have this fix in bionic, please.

Status changed to 'Confirmed' because the bug affects multiple users.

A new bind has been pushed to bionic (1:9.11.3+dfsg-1ubuntu1.2). This is newer than bind9 in ppa:freeipa/ppa, but does not contain the fix for this bug. Therefore, bind9 upgrade should be prevented by helding the ppa package, or bind9-pkcs11 will stop working.

I'll take care of this for bionic.

Uploaded to bionic unapproved, waiting for SRU team's approval.

@ahasenack When you said "Uploaded to bionic unapproved", did you mean 1:9.11.3+dfsg-1ubuntu1.3?

It's this one:
bind9 (1:9.11.3+dfsg-1ubuntu1.3) bionic; urgency=medium

  [ Karl Stenerud ]
  * d/p/skip-rtld-deepbind-for-dyndb.diff: fix named-pkcs11 crashing on
    startup. Thanks to Petr Menšík <email address hidden> (LP: #1769440)

 -- Andreas Hasenack <email address hidden> Wed, 10 Oct 2018 14:33:34 -0300

It's still in the unapproved queue:
https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=bind9

Hello keestux, or anyone else affected,

Accepted bind9 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/bind9/1:9.11.3+dfsg-1ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I'm having a hard time reproducing the bug in bionic nowadays. It feels like a timing issue, because right after it complains about a connection refused, I can connect just fine:

(...)
  [24/28]: migrating certificate profiles to LDAP
  [error] NetworkError: cannot connect to 'https://bionic-freeipa.example.internal:8443/ca/rest/account/logout': [Errno 111] Connection refused
ipapython.admintool: ERROR cannot connect to 'https://bionic-freeipa.example.internal:8443/ca/rest/account/logout': [Errno 111] Connection refused
ipapython.admintool: ERROR The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

root@bionic-freeipa:~# telnet localhost 8443
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

I'll keep trying, but it would help if others could also try.

I've installed proposed packages for bind. Now service is working for me.
After install proposed package:
# ipactl restart
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: STOPPED
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

Package version 1:9.11.3+dfsg-1ubuntu1.3 in -proposed also works for me.

This bug was fixed in the package bind9 - 1:9.11.3+dfsg-1ubuntu1.3

---------------
bind9 (1:9.11.3+dfsg-1ubuntu1.3) bionic; urgency=medium

  [ Karl Stenerud ]
  * d/p/skip-rtld-deepbind-for-dyndb.diff: fix named-pkcs11 crashing on
    startup. Thanks to Petr Menšík <email address hidden> (LP: #1769440)

 -- Andreas Hasenack <email address hidden> Wed, 10 Oct 2018 14:33:34 -0300

The verification of the Stable Release Update for bind9 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hi, sorry for bumping this bug, but I'm using the latest version of bind (1:9.11.3+dfsg-1ubuntu1.3) for Bionic, which should contain the fix, but named-pkcs11 is still crashing for me.

It crashes at:
26-Dec-2018 12:11:07.639 ../../../lib/isc-pkcs11/md5.c:93: fatal error:
26-Dec-2018 12:11:07.639 RUNTIME_CHECK(pk11_get_session(ctx, OP_DIGEST, isc_boolean_true, isc_boolean_false, isc_boolean_false, ((void *)0), 0) == 0) failed

Thanks in advance.