ntpd crashes using GPSD_JSON clock

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal

# apt-cache policy ntp
ntp:
  Installed: 1:4.2.8p12+dfsg-3ubuntu4
  Candidate: 1:4.2.8p12+dfsg-3ubuntu4
  Version table:
 *** 1:4.2.8p12+dfsg-3ubuntu4 500
        500 http://ports.ubuntu.com/ubuntu-ports focal/universe arm64 Packages
        100 /var/lib/dpkg/status

# arch
aarch64

I am trying to use gpsd as a time source for ntp. When I add this to ntp.conf:

server 127.127.46.0 minpoll 4 maxpoll 4 noselect
fudge 127.127.46.0 time1 0.0 time2 0.0 refid GPS

Then ntp will crash shortly after starting although the amount of time the daemon runs for is not constant. Running the normal ntpd command and adding -n and -d the last few lines of output are:

 8 Apr 14:34:14 ntpd[88025]: GPSD_JSON(0): GPSD revision=3.20 release=3.20 protocol=3.14
refclock_transmit: at 3 127.127.46.0
refclock_receive: at 3 127.127.46.0
event at 3 GPSD_JSON(0) 8014 84 reachable
refclock_sample: n 1 offset -0.221798 disp 0.000000 jitter 0.000000
filegen 2 3826877655
clock_filter: n 1 off -0.221798 del 0.000000 dsp 7.937500 jit 0.000000
 8 Apr 14:34:16 ntpd[88025]: work_thread.c:219: INSIST(((void *)0) == c->workitems[c->head_workitem % c->workitems_alloc]) failed
 8 Apr 14:34:16 ntpd[88025]: exiting (due to assertion failure)
Aborted (core dumped)

gpsd is running and has a fix, the PPS and SHM ntp clocks are working with ntp.

# ntpq -crv -pn
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd 4.2.8p12@1.3728-o (1)", processor="aarch64",
system="Linux/5.4.0-1032-raspi", leap=00, stratum=4, precision=-22,
rootdelay=30.245, rootdisp=79.073, refid=15.39.164.186,
reftime=e419899f.4a3ad96f Thu, Apr 8 2021 14:54:39.289,
clock=e4198a33.8e7058bf Thu, Apr 8 2021 14:57:07.556, peer=13164, tc=6,
mintc=3, offset=0.638826, frequency=-2.295, sys_jitter=0.034515,
clk_jitter=0.241, clk_wander=0.065, tai=37, leapsec=201701010000,
expire=202112280000

     remote refid st t when poll reach delay offset jitter
==============================================================================
 127.127.22.0 .PPS. 0 l - 16 377 0.000 -0.869 0.028
 127.127.28.0 .SHM. 15 l 15 16 377 0.000 -119.64 5.299
 pool.example.com.POOL. 16 p - 64 0 0.000 0.000 0.000
*<ipv6> 176.58.109.199 3 u 15 64 377 0.415 0.854 0.179
+<ipv4> 176.58.109.199 3 u 18 64 377 0.567 0.825 0.140