LXD Storage pool on an NFS share strange user mapping

Hi

Ref: 18.04 LTS

I originally posted this problem on JUJU discourse (https://discourse.jujucharms.com/t/lxc-with-nfs-storage-pool-fail-bootstrap-and-deploy/1654) but then move to LXD issue (https://github.com/lxc/lxd/issues/5962).

The problem is when an LXD container has a storage pool on an NFS share is dir mode creating files either touch, vim, ... the UID of the file is set to nobody but chown of the file to root fixes the problem.

I am in the process of testing 19.04 and 19.10 as well.

Matthew

Here is the report from LXD issue.

================================

 Required information

Distribution:Ubuntu
Distribution version: 18.04 LTS
The output of "lxc info" or if that fails:
config:
core.https_address: 10.0.4.1:8443
core.trust_password: true
maas.api.key: ****
maas.api.url: http://*****:5240/MAAS
api_extensions:
storage_zfs_remove_snapshots
container_host_shutdown_timeout
container_stop_priority
container_syscall_filtering
auth_pki
container_last_used_at
etag
patch
usb_devices
https_allowed_credentials
image_compression_algorithm
directory_manipulation
container_cpu_time
storage_zfs_use_refquota
storage_lvm_mount_options
network
profile_usedby
container_push
container_exec_recording
certificate_update
container_exec_signal_handling
gpu_devices
container_image_properties
migration_progress
id_map
network_firewall_filtering
network_routes
storage
file_delete
file_append
network_dhcp_expiry
storage_lvm_vg_rename
storage_lvm_thinpool_rename
network_vlan
image_create_aliases
container_stateless_copy
container_only_migration
storage_zfs_clone_copy
unix_device_rename
storage_lvm_use_thinpool
storage_rsync_bwlimit
network_vxlan_interface
storage_btrfs_mount_options
entity_description
image_force_refresh
storage_lvm_lv_resizing
id_map_base
file_symlinks
container_push_target
network_vlan_physical
storage_images_delete
container_edit_metadata
container_snapshot_stateful_migration
storage_driver_ceph
storage_ceph_user_name
resource_limits
storage_volatile_initial_source
storage_ceph_force_osd_reuse
storage_block_filesystem_btrfs
resources
kernel_limits
storage_api_volume_rename
macaroon_authentication
network_sriov
console
restrict_devlxd
migration_pre_copy
infiniband
maas_network
devlxd_events
proxy
network_dhcp_gateway
file_get_symlink
network_leases
unix_device_hotplug
storage_api_local_volume_handling
operation_description
clustering
event_lifecycle
storage_api_remote_volume_handling
nvidia_runtime
candid_authentication
candid_config
candid_config_key
usb_optional_vendorid
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
tls
environment:
addresses:
10.0.4.1:8443
architectures:
x86_64
i686
certificate: |
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
certificate_fingerprint:
driver: lxc
driver_version: 3.0.3
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.15.0-54-generic
server: lxd
server_pid: 30277
server_version: 3.0.3
storage: dir
storage_version: "1"
server_clustered: true
server_name: m-node-01
project: ""
UID mapping incorrect when creating a file.

When touching or creating a file with vim (or any other method) is created with the UID on nobody (65534) but other files are all correct the startup. The correct UID can be set by chown of the file

Steps to reproduce

Create container (lxc launch -s nfs ubuntu:18.04)
Access container ( lxc exec modest-ladybug -- sudo /bin/bash )
touch file (touch test)
list dir (ls -la)
root@modest-ladybug:~# ls -la
total 50
drwx------ 3 root root 6 Jul 15 21:18 .
drwxr-xr-x 22 root root 22 Jun 27 16:28 ..
-rw-r--r-- 1 root root 3106 Apr 9 2018 .bashrc
-rw-r--r-- 1 root root 148 Aug 17 2015 .profile
drwx------ 2 root root 3 Jul 15 21:17 .ssh
-rw-r--r-- 1 nobody root 0 Jul 15 21:18 test
Change own (chown root:root test)
root@modest-ladybug:~# ls -la
total 50
drwx------ 3 root root 6 Jul 15 21:18 .
drwxr-xr-x 22 root root 22 Jun 27 16:28 ..
-rw-r--r-- 1 root root 3106 Apr 9 2018 .bashrc
-rw-r--r-- 1 root root 148 Aug 17 2015 .profile
drwx------ 2 root root 3 Jul 15 21:17 .ssh
-rw-r--r-- 1 root root 0 Jul 15 21:18 test
Information to attach

  Container log (lxc info NAME --show-log)
Name: modest-ladybug
Location: m-node-02
Remote: unix://
Architecture: x86_64
Created: 2019/07/15 21:15 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 24080
Ips:
eth0: inet 10.1.128.8 veth68YBCW
eth0: inet6 fe80::216:3eff:fee4:bcc0 veth68YBCW
lo: inet 127.0.0.1
lo: inet6 ::1
Resources:
Processes: 24
CPU usage:
CPU usage (in seconds): 9
Memory usage:
Memory (current): 220.18MB
Memory (peak): 279.36MB
Network usage:
eth0:
Bytes received: 832.51kB
Bytes sent: 14.13kB
Packets received: 5312
Packets sent: 176
lo:
Bytes received: 1.42kB
Bytes sent: 1.42kB
Packets received: 16
Packets sent: 16
Log:

lxc modest-ladybug 20190715211703.342 WARN conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - Failed to unmount old devpts instance

  Container configuration (lxc config show NAME --expanded)
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20190627.1)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: "20190627.1"
image.version: "18.04"
volatile.base_image: 6ae1c6e92017402f1aee655fa8d785ee9d2337a3369d76115cecad5e7a303e07
volatile.eth0.hwaddr: 00:16:3e:e4:bc:c0
volatile.idmap.base: "0"
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.power: RUNNING
devices:
eth0:
maas.subnet.ipv4: system-subnet
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: nfs
type: disk
ephemeral: false
profiles:

default
stateful: false
description: ""

  Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
t=2019-07-15T08:50:07+0000 lvl=info msg="Updating images"
t=2019-07-15T08:50:07+0000 lvl=info msg="Done updating images"
t=2019-07-15T14:50:07+0000 lvl=info msg="Pruning expired images"
t=2019-07-15T14:50:07+0000 lvl=info msg="Done pruning expired images"
t=2019-07-15T14:50:07+0000 lvl=info msg="Expiring log files"
t=2019-07-15T14:50:07+0000 lvl=info msg="Done expiring log files"
t=2019-07-15T14:50:07+0000 lvl=info msg="Updating instance types"
t=2019-07-15T14:50:07+0000 lvl=info msg="Done updating instance types"
t=2019-07-15T14:50:07+0000 lvl=info msg="Updating images"
t=2019-07-15T14:50:07+0000 lvl=info msg="Done updating images"
t=2019-07-15T17:42:57+0000 lvl=warn msg="Detected poll(POLLNVAL) event."
t=2019-07-15T18:01:57+0000 lvl=warn msg="Failed to get current cluster nodes: failed to begin transaction: failed to receive response: failed to receive header: failed to receive header: EOF"
t=2019-07-15T18:01:58+0000 lvl=warn msg="Failed to get current cluster nodes: failed to begin transaction: failed to receive response: failed to receive header: failed to receive header: EOF"
t=2019-07-15T18:02:42+0000 lvl=warn msg="Failed to get events from node 10.0.4.2:8443: Unable to connect to: 10.0.4.2:8443"
t=2019-07-15T18:05:15+0000 lvl=info msg="Received 'terminated signal', exiting"
t=2019-07-15T18:05:15+0000 lvl=info msg="Starting shutdown sequence"
t=2019-07-15T18:05:15+0000 lvl=info msg="Stopping REST API handler:"
t=2019-07-15T18:05:15+0000 lvl=info msg=" - closing socket" socket=10.0.4.1:8443
t=2019-07-15T18:05:15+0000 lvl=info msg=" - closing socket" socket=/var/lib/lxd/unix.socket
t=2019-07-15T18:05:15+0000 lvl=info msg="Stopping /dev/lxd handler"
t=2019-07-15T18:05:15+0000 lvl=info msg=" - closing socket" socket=/var/lib/lxd/devlxd/sock
t=2019-07-15T18:05:15+0000 lvl=info msg="Closing the database"
t=2019-07-15T18:05:15+0000 lvl=info msg="Stopping REST API handler:"
t=2019-07-15T18:05:15+0000 lvl=info msg="Stopping /dev/lxd handler"
t=2019-07-15T18:05:15+0000 lvl=info msg="Stopping REST API handler:"
t=2019-07-15T18:05:15+0000 lvl=info msg="Stopping /dev/lxd handler"
t=2019-07-15T18:05:15+0000 lvl=info msg="Saving simplestreams cache"
t=2019-07-15T18:05:15+0000 lvl=info msg="Saved simplestreams cache"
t=2019-07-15T18:05:15+0000 lvl=info msg="LXD 3.0.3 is starting in normal mode" path=/var/lib/lxd
t=2019-07-15T18:05:15+0000 lvl=info msg="Kernel uid/gid map:"
t=2019-07-15T18:05:15+0000 lvl=info msg=" - u 0 0 4294967295"
t=2019-07-15T18:05:15+0000 lvl=info msg=" - g 0 0 4294967295"
t=2019-07-15T18:05:15+0000 lvl=info msg="Configured LXD uid/gid map:"
t=2019-07-15T18:05:15+0000 lvl=info msg=" - u 0 100000 1000000000"
t=2019-07-15T18:05:15+0000 lvl=info msg=" - g 0 100000 1000000000"
t=2019-07-15T18:05:15+0000 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
t=2019-07-15T18:05:15+0000 lvl=info msg="Kernel features:"
t=2019-07-15T18:05:15+0000 lvl=info msg=" - netnsid-based network retrieval: no"
t=2019-07-15T18:05:15+0000 lvl=info msg=" - unprivileged file capabilities: yes"
t=2019-07-15T18:05:15+0000 lvl=info msg="Initializing local database"
t=2019-07-15T18:05:15+0000 lvl=info msg="Starting /dev/lxd handler:"
t=2019-07-15T18:05:15+0000 lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock
t=2019-07-15T18:05:15+0000 lvl=info msg="REST API daemon:"
t=2019-07-15T18:05:15+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/lib/lxd/unix.socket
t=2019-07-15T18:05:15+0000 lvl=info msg=" - binding TCP socket" socket=10.0.4.1:8443
t=2019-07-15T18:05:15+0000 lvl=info msg="Initializing global database"
t=2019-07-15T18:05:16+0000 lvl=info msg="Initializing storage pools"
t=2019-07-15T18:05:16+0000 lvl=info msg="Initializing networks"
t=2019-07-15T18:05:16+0000 lvl=info msg="Pruning leftover image files"
t=2019-07-15T18:05:16+0000 lvl=info msg="Done pruning leftover image files"
t=2019-07-15T18:05:16+0000 lvl=info msg="Loading daemon configuration"
t=2019-07-15T18:05:16+0000 lvl=info msg="Pruning expired images"
t=2019-07-15T18:05:16+0000 lvl=info msg="Done pruning expired images"
t=2019-07-15T18:05:16+0000 lvl=info msg="Expiring log files"
t=2019-07-15T18:05:16+0000 lvl=info msg="Done expiring log files"
t=2019-07-15T18:05:16+0000 lvl=info msg="Updating instance types"
t=2019-07-15T18:05:16+0000 lvl=info msg="Done updating instance types"
t=2019-07-15T18:05:16+0000 lvl=info msg="Updating images"
t=2019-07-15T18:05:16+0000 lvl=info msg="Done updating images"
t=2019-07-15T18:05:17+0000 lvl=info msg="Connected to MAAS controller" url=http://****:5240/MAAS
t=2019-07-15T18:09:35+0000 lvl=info msg="Stopping container" action=stop created=2019-07-14T21:04:31+0000 ephemeral=false name=juju-594876-10 stateful=false used=2019-07-14T21:04:39+0000
t=2019-07-15T18:09:36+0000 lvl=info msg="Stopped container" action=stop created=2019-07-14T21:04:31+0000 ephemeral=false name=juju-594876-10 stateful=false used=2019-07-14T21:04:39+0000
t=2019-07-15T18:09:36+0000 lvl=info msg="Deleting container" created=2019-07-14T21:04:31+0000 ephemeral=false name=juju-594876-10 used=2019-07-14T21:04:39+0000
t=2019-07-15T18:09:40+0000 lvl=info msg="Deleted container" created=2019-07-14T21:04:31+0000 ephemeral=false name=juju-594876-10 used=2019-07-14T21:04:39+0000
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jul 16 07:51 seq
 crw-rw---- 1 root audio 116, 33 Jul 16 07:51 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 18.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: Dell Inc. PowerEdge R710
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
Package: linux (not installed)
PciMultimedia:

ProcFB: 0 mgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-54-generic root=UUID=46cbca7e-01a0-4fef-badb-d768f679fcbe ro
ProcVersionSignature: User Name 4.15.0-54.58-generic 4.15.18
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-54-generic N/A
 linux-backports-modules-4.15.0-54-generic N/A
 linux-firmware 1.173.8
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic uec-images
Uname: Linux 4.15.0-54-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 02/19/2018
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 6.5.0
dmi.board.name: 0HYPX2
dmi.board.vendor: Dell Inc.
dmi.board.version: A02
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr6.5.0:bd02/19/2018:svnDellInc.:pnPowerEdgeR710:pvr:rvnDellInc.:rn0HYPX2:rvrA02:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R710
dmi.sys.vendor: Dell Inc.
---
ProblemType: Bug
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
CRDA: N/A
DistroRelease: Ubuntu 18.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: Dell Inc. PowerEdge R710
NonfreeKernelModules: cfg80211 binfmt_misc veth ebtable_filter ebtables zfs zunicode zavl icp zcommon znvpair spl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache bridge stp llc intel_powerclamp coretemp ipmi_ssif kvm_intel kvm irqbypass intel_cstate i7core_edac dcdbas gpio_ich input_leds joydev serio_raw lpc_ich shpchp ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter mac_hid sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mlx4_en mlx4_ib ptp ib_core pps_core crct10dif_pclmul crc32_pclmul hid_generic ghash_clmulni_intel mgag200 i2c_algo_bit pcbc ttm drm_kms_helper aesni_intel syscopyarea mpt3sas aes_x86_64 sysfillrect sysimgblt crypto_simd fb_sys_fops glue_helper usbhid mlx4_core raid_class cryptd psmouse drm hid pata_acpi devlink scsi_transport_sas bnx2 wmi
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB: 0 mgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-54-generic root=UUID=46cbca7e-01a0-4fef-badb-d768f679fcbe ro
ProcVersionSignature: Ubuntu 4.15.0-54.58-generic 4.15.18
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-54-generic N/A
 linux-backports-modules-4.15.0-54-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic uec-images
Uname: Linux 4.15.0-54-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

WifiSyslog:

_MarkForUpload: True
dmi.bios.date: 02/19/2018
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 6.5.0
dmi.board.name: 0HYPX2
dmi.board.vendor: Dell Inc.
dmi.board.version: A02
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr6.5.0:bd02/19/2018:svnDellInc.:pnPowerEdgeR710:pvr:rvnDellInc.:rn0HYPX2:rvrA02:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R710
dmi.sys.vendor: Dell Inc.