Cannot start docker container on ubuntu 21.10 server for raspberry pi

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1947601

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

After downgrading the linux kernel to the latest version shipped with ubuntu 21.04 and rebooting, executing a docker container works in ubuntu 21.10 for 64-bit Raspberry Pi.

The downgrading can be achieved like this:

1) Add the hirsute apt repositories to /etc/apt/sources.list
2) sudo apt install linux-image-5.11.0-1021-raspi
3) sudo flash-kernel --force 5.11.0-1021-raspi
4) Reboot.

This indicates that there really is a problem with the current kernel in 21.10 that prevents execution of docker containers.

The 32 bit raspberry pi image for ubuntu 21.10 is also affected.

64-bit PC architecture (amd64) is not affected.

The vent kernel module is not installed. If you do:
sudo apt install linux-modules-extra-raspi

This will install the module and docker will work.

Good to know that there is a better workaround than downgrading the kernel.

I do not see any module named "vent" in the modules-extra package for raspberry pi. I think you meant "veth", right?

Why did this module change from standard modules to modules-extra between linux 5.11 and 5.13? Only raspi seems to be affected, my 21.10 test machine for PC does not have modules-extra installed, and docker still works there.

See also https://packages.ubuntu.com/search?searchon=contents&keywords=veth.ko&mode=exactfilename&suite=impish&arch=any for the package containing the veth module on the different architectures. Compare with the situation for 21.04: https://packages.ubuntu.com/search?searchon=contents&keywords=veth.ko&mode=exactfilename&suite=hirsute&arch=any (no modules-extra anywhere).

Can this module please move back to the standard module package also for the raspberry pi kernels?

linux-modules-extra-raspi was added in Impish to attempt to reduce the size of the initrd (and general memory usage) for most users. The update-initramfs tool is configured (by default) to include "most" modules when building the initrd; without linux-modules-extra-raspi installed this results in a ~19MB initrd (again, assuming the default zstd compression). With linux-modules-extra-raspi installed this doubles in size to 38MB, adding several seconds to the boot time and loading a lot of modules that, for most users, are never needed.

Incidentally, a similar split exists in a couple of other places (linux-modules-extra-aws, linux-modules-extra-oracle). If you want to install this package by default on new images, I'd recommend having a look at the "user-data" cloud-init configuration file on the boot partition. This has a few examples in it, including installing extra packages (like linux-modules-extra-raspi) on first boot.

I understand the motivation. Reducing boot time by reducing the number of kernel modules is something I can understand.

However, the original problem is not solved. I had to invest >10 hours, trying to get docker containers to work with ubuntu 21.10 on Raspberry Pi.

Why was a solution for running docker containers not mentioned in the Ubuntu 21.10 release notes?
Why have you, Dave, not immediately told me of the solution (install modules-extra!) as soon as you saw this bug report?
Why has the docker.io package not added a dependency on the modules-extra package? It clearly does not work without this package on raspi.
Why was the veth kernel module moved to modules-extra? You are talking about 19MB space savings of compressed initrd, but this module is only 25k, uncompressed! And on ubuntu servers, Docker certainly is one of the main use cases.

I feel that there needs to be a fix other than invalidating my >10 hours of my work by invalidating this bug report.

Coming back to boot time savings: IMO that's a thing for desktops. But for servers? I mean, servers are meant to run uninterrupted until there is a kernel update. A few seconds more per reboot therefore does not eat away from the uptime. Your server images, therefore, should have modules-extra installed by default.

We use ubuntu instead of raspi-os on our ARM build servers only **because** we need to run Docker for ARM 32 and 64 bits. Please find another solution for this cannot-run-docker-out-of-the-box problem for your server image than letting all your users in the dark about why they cannot run docker anymore in 21.10.

For what it is worth, I agree with ianmacs. I spent 2 days googling to find a solution to the problem. I'm using the Ubuntu Desktop 64 bit version. Just because I like to have a desktop (just in case). But have been thinking about moving all my containers to the server version. Might do that when the next LTS version is released. So, I just do an in place upgrade from 21.04 to 21.10 and then docker would not work any more. I could not find anyone else reporting the issue that I was having with the VETH module. I use the official Docker repository for getting docker. As of yesterday, they have not added the Impish repository (hopefully, I'm using the right terms here). If someone can give me a good reason to just sick with the version of docker that is in the official Ubuntu repository, maybe I will just use that.

I don't know what the solution is. If you make the modules-extra a dependency on docker.io, that won't help me if I'm getting docker from docker. All I know is that a lot of people are running docker on the PI. I guess they are just not using UBUNTU OS on the PI. I just like what UBUNTU team is doing, so I want to stick with that OS.

Does it make sense to just move that one module into the kernel image? (Not sure if any other modules are need for docker).

Thanks for all your hard work, I do appreciate it.

Zorin

> Why was a solution for running docker containers not mentioned in the Ubuntu 21.10 release notes?

My apologies -- that's my oversight, which I have now corrected, including a note about Docker. That said, had I remembered to include it last week I can't say I would've mentioned Docker. It's a package in universe and thus community supported. As far as I'm aware it's not something we directly test with the Pi images.

> Why have you, Dave, not immediately told me of the solution (install modules-extra!) as soon as you saw this bug report?

Because by the time I noticed this bug report (a few hours ago), I'd also read the comments and noted you'd already been provided with the solution (i.e. install linux-modules-extra-raspi). All I could reasonably add to that was a means to do so implicitly on the first boot (my cloud-init user-data suggestion).

> Why has the docker.io package not added a dependency on the modules-extra package? It clearly does not work without this package on raspi.

While it's possible to add per-architecture dependencies to debs (e.g. Depends: linux-modules-extra-raspi[armhf arm64]), that's not specific enough for this case. It'd be incorrect for Docker operating on an AWS arm64 instance, for example.

It *could* be added as Suggests: linux-modules-extra-raspi[armhf arm64] (I *think* per-architecture specs are valid in Suggests as well as the various Depends variations). Again, this wouldn't be strictly correct (it's evidently a hard dependency on the raspi platform), but it would at least hint to users when installing docker.io that something extra might be needed. It couldn't be added as Recommends: because, again, that would result in its implicit installation on other non-pi armhf/arm64 platforms.

> Why was the veth kernel module moved to modules-extra? You are talking about 19MB space savings of compressed initrd, but this module is only 25k, uncompressed! And on ubuntu servers, Docker certainly is one of the main use cases.

We only have sufficient resources for one -raspi kernel package, hence that package (and its corresponding modules package(s)) needs to cover both server and desktop use-cases. In this case, the "veth" module, while obviously vital for Docker, is clearly an optional component if seen from the perspective of covering both server and desktop cases; to most desktop users it's redundant.

> I feel that there needs to be a fix other than invalidating my >10 hours of my work by invalidating this bug report.

I'm sorry it's taken you >10 hours to discover the underlying cause, and a solution to it, and that's at least partly my fault for forgetting to include the existence of linux-modules-extra-raspi in the Impish release notes. However, I'm afraid in the case of the linux-raspi package, this certainly is invalid: it's operating as intended.

I could re-point this bug to the docker.io package, if you'd prefer, but (for reasons given above) I'm not sure there's a valid solution there either?

> Coming back to boot time savings: IMO that's a thing for desktops. But for servers? I mean, servers are meant to run uninterrupted until there is a kernel update. A few seconds more per reboot therefore does...

Thank you very much, Dave, for taking the time to consider my input.

> I'm not convinced changing that to "sudo apt install docker.io
> linux-modules-extra-raspi" is such a burden?

For me no longer, since I know about it now. For all other ubuntu
users out there, who do not know about this workaround yet: yes, very
much so. The promise of apt is, that the user only has to know which
software they want to install, and apt will automatically install the
dependencies. This promise is broken here. Also consider the
users who will upgrade from 20.04 to 22.04 in half a year, and who
will also at first be surprised that their usual method of running
docker will no longer work.

Thanks for updating the release notes. Please note that there is a
typo that you may want to correct (rapsi -> raspi) in the workaround
command.

With regard to a possible fix:

I have checked which kernel modules are needed from the current
modules-extra package to run my docker projects: It really is only
veth, no other module is missing from the main modules package.

You say that rarely-used modules got transferred to the
modules-extra package. I would like to ask again that veth be
reconsidered as more-often-than-rarely used because it is essential
for docker, docker being quite popular on raspberry pi servers,
and because it is so small that it does not interfere with your
boot time acceleration goal.

Another possible fix would be fixing the dependencies. This is
more complicated than moving veth back to main modules, but I'm
sure it can be done in some way. I'm not an apt dependency expert,
but isn't it possible to have docker.io depend on a virtual package
veth, and then have different parts of the ARM kernel module
packages provide this veth module. I'm sure there is also a ton
of other possibilities, but all of them more complicated than
simply moving the module back.

As Dave mentioned, it's a new package in Impish and is certainly not perfect. I have no problems moving that module to linux-modules.

FWIW, I missed this bug report because it was initially logged against the linux package instead of linux-raspi.

Also, we do get a lot of tickets and sometimes things fall through the cracks for which I apologize. If you feel you don't get enough attention, you can always try to reach us on IRC libera.chat (channel #ubuntu-kernel for the kernel team for example).

This bug is awaiting verification that the linux-raspi/5.13.0-1011.13 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-impish' to 'verification-done-impish'. If the problem still exists, change the tag 'verification-needed-impish' to 'verification-failed-impish'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Installed linux-raspi/5.13.0-1011.13 from impish-proposed and verified that 'docker hello-world' succeeds.

Thanks for this fix! I have upgraded linux-raspi to the version now on impish-proposed, uninstalled linux-modules-extra-raspi, rebooted and all my docker projects still work.

Now it would be nice if this package can make the step from impish-proposed to impish-updates so that all users automatically profit from the update. In any case, this change is a good fix for the upcoming 22.04 LTS release.

For the time being, this is a possible way for 21.10 users to upgrade to the new linux-raspi version in order to work around the original problem without having to install the modules-extra package:

$ echo deb http://ports.ubuntu.com/ubuntu-ports impish-proposed main | sudo tee /etc/apt/sources.list.d/proposed.list
$ sudo apt update
$ sudo apt install linux-raspi
$ sudo rm /etc/apt/sources.list.d/proposed.list
$ sudo apt update
$ sudo reboot

This bug was fixed in the package linux-raspi - 5.13.0-1011.13

---------------
linux-raspi (5.13.0-1011.13) impish; urgency=medium

  * impish/linux-raspi: 5.13.0-1011.13 -proposed tracker (LP: #1949737)

  * Cannot start docker container on ubuntu 21.10 server for raspberry pi
    (LP: #1947601)
    - [Packaging] raspi: Add veth module to linux-modules package

  * /dev/ttyUSB? port lost on upgrading to 21.10 (LP: #1947723)
    - [Packaging] raspi: Add USB serial modules to linux-modules package

  * Missing modules in linux-modules (LP: #1949370)
    - [Packaging] raspi: Add additional modules to linux-modules package

  [ Ubuntu: 5.13.0-22.22 ]

  * impish/linux: 5.13.0-22.22 -proposed tracker (LP: #1949740)
  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2021.11.08)
  * ebpf: bpf_redirect fails with ip6 gre interfaces (LP: #1947164)
    - net: handle ARPHRD_IP6GRE in dev_is_mac_header_xmit()
  * require CAP_NET_ADMIN to attach N_HCI ldisc (LP: #1949516)
    - Bluetooth: hci_ldisc: require CAP_NET_ADMIN to attach N_HCI ldisc
  * CVE-2021-3744 // CVE-2021-3764
    - crypto: ccp - fix resource leaks in ccp_run_aes_gcm_cmd()
  * ppc64 BPF JIT mod by 1 will not return 0 (LP: #1948351)
    - powerpc/bpf: Fix BPF_MOD when imm == 1
  * Fix Screen freeze after resume from suspend with iGPU [1002:6987]
    (LP: #1949050)
    - drm/amdgpu: reenable BACO support for 699F:C7 polaris12 SKU
    - drm/amdgpu: add missing cleanups for Polaris12 UVD/VCE on suspend
    - drm/amdgpu: Fix crash on device remove/driver unload
  * Intel I225-IT ethernet controller: igc: probe of 0000:02:00.0 failed with
    error -1 (LP: #1945576)
    - igc: Remove _I_PHY_ID checking
    - igc: Remove phy->type checking
  * Fail to detect audio output from external monitor (LP: #1948767)
    - ALSA: hda: intel: Allow repeatedly probing on codec configuration errors
  * Drop "UBUNTU: SAUCE: cachefiles: Page leaking in
    cachefiles_read_backing_file while vmscan is active" (LP: #1947709)
    - Revert "UBUNTU: SAUCE: cachefiles: Page leaking in
      cachefiles_read_backing_file while vmscan is active"
  * rtw89 kernel module for Realtek 8852 wifi is missing (LP: #1945967)
    - rtw89: add Realtek 802.11ax driver
    - rtw89: Remove redundant check of ret after call to rtw89_mac_enable_bb_rf
    - rtw89: fix return value check in rtw89_cam_send_sec_key_cmd()
    - rtw89: remove unneeded semicolon
    - [Config] RTW89=m
  * Impish update: upstream stable patchset 2021-11-03 (LP: #1949636)
    - mm: fix uninitialized use in overcommit_policy_handler
    - usb: gadget: r8a66597: fix a loop in set_feature()
    - usb: dwc2: gadget: Fix ISOC flow for BDMA and Slave
    - usb: dwc2: gadget: Fix ISOC transfer complete handling for DDMA
    - usb: musb: tusb6010: uninitialized data in tusb_fifo_write_unaligned()
    - cifs: fix incorrect check for null pointer in header_assemble
    - xen/x86: fix PV trap handling on secondary processors
    - usb-storage: Add quirk for ScanLogic SL11R-IDE older than 2.6c
    - USB: serial: cp210x: add ID for GW Instek GDM-834x Digital Multimeter
    - USB: cdc-acm: fix minor-number release
    - Rev...

Hey,
Its not possible install linux-modules-extra-raspi in chroot, because it activates flash-kernel which fails.
I also tried to bypass flash-kernel by moving /sbin/flash-kernel aside before installing, but that then the issue does not get fixed.

This means that a number of distros I maintain can't have docker running correctly.

Related: https://github.com/guysoft/CustomPiOS/issues/149

@guysoft, this seems like a different issue so please file a new bug report.

Done, see: https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/1955353

This bug was fixed in the package linux-raspi - 5.15.0-1002.2

---------------
linux-raspi (5.15.0-1002.2) jammy; urgency=medium

  * jammy/linux-raspi: 5.15.0-1002.2 -proposed tracker (LP: #1958834)

  * Packaging resync (LP: #1786013)
    - [Packaging] update Ubuntu.md
    - debian/dkms-versions -- update from kernel-versions (main/master)

  * Kernel fails to boot in ScalingStack (LP: #1959102)
    - [Config] raspi: Set VIRTIO_PCI=m
    - [Config] raspi: Set ACPI=y

  * jammy/linux-raspi: Update to upstream raspberrypi rpi-5.15.y (2022-01-24)
    (LP: #1958854)
    - brcmfmac: firmware: Fix crash in brcm_alt_fw_path
    - ARM: dts: Remove VL805 USB node from CM4 dts
    - mfd: simple-mfd-i2c: Add configuration for RPi POE HAT
    - pwm: raspberrypi-poe: Add option of being created by MFD or FW
    - power: rpi-poe: Drop CURRENT_AVG as it is not hardware averaged
    - power: rpi-poe: Add option of being created by MFD or FW
    - defconfigs: Add MFD_RASPBERRYPI_POE_HAT to Pi defconfigs.
    - dtoverlays: Add option for PoE HAT to use Linux I2C instead of FW.
    - drivers: bcm2835_unicam: Disable trigger mode operation
    - arm: Remove spurious .fnend directive
    - drm/vc4: Fix deadlock on DSI device attach error
    - drm/vc4: dsi: Correct max divider to 255 (not 7)
    - defconfig: Add BACKLIGHT_PWM to bcm2709 and bcmrpi defconfigs
    - dtoverlays: Add pwm backlight option to vc4-kms-dpi-generic
    - dtoverlays: Correct [h|v]sync_invert config in vc4-kms-dpi-generic
    - ARM: dts: BCM2711 AON_INTR2 generates IRQ edges
    - update rpi-display-overlay.dts pins for 5.10+

  [ Ubuntu: 5.15.0-18.18 ]

  * jammy/linux: 5.15.0-18.18 -proposed tracker (LP: #1958638)
  * CVE-2021-4155
    - xfs: map unwritten blocks in XFS_IOC_{ALLOC, FREE}SP just like fallocate
  * CVE-2022-0185
    - SAUCE: vfs: test that one given mount param is not larger than PAGE_SIZE
  * [UBUNTU 20.04] KVM hardware diagnose data improvements for guest kernel -
    kernel part (LP: #1953334)
    - KVM: s390: add debug statement for diag 318 CPNC data
  * OOB write on BPF_RINGBUF (LP: #1956585)
    - SAUCE: bpf: prevent helper argument PTR_TO_ALLOC_MEM to have offset other
      than 0
  * Miscellaneous Ubuntu changes
    - [Config] re-enable shiftfs
    - [SAUCE] shiftfs: support kernel 5.15
    - [Config] update toolchain versions
  * Miscellaneous upstream changes
    - vfs: fs_context: fix up param length parsing in legacy_parse_param

linux-raspi (5.15.0-1001.1) jammy; urgency=medium

  * Missing overlays/README (LP: #1954757)
    - SAUCE: Install overlays/README

  * dtoverlay=uart4 breaks Raspberry Pi 4B boot (LP: #1875454)
    - SAUCE: arm: dts: Add 'brcm,bcm2835-pl011' compatible for uart2-5

  * jammy/linux-raspi: Update to upstream raspberrypi rpi-5.15.y (2022-01-14)
    (LP: #1958146)
    - clk: bcm-2835: Pick the closest clock rate
    - clk: bcm-2835: Remove rounding up the dividers
    - drm/vc4: hdmi: Set a default HSM rate
    - drm/vc4: hdmi: Move the HSM clock enable to runtime_pm
    - drm/vc4: hdmi: Make sure the controller is powered in detect
    - drm/vc4: hdmi: Make sure the controller is powered up during bind
    - drm/vc4: hdmi: Rework the ...