Related: https://bugzilla.redhat.com/show_bug.cgi?id=2021203
Description of problem:
When forking a pty as regular user, we get a failure to read/write on /dev/ptmx since it's just a symlink to /dev/pts/ptmx.
In fedora, /dev/ptmx is its own node.
[1] centos 9
[2] fedora
[3] strace
[4] reproducer script
After doing further investivation, Openshift product has seen a similar issue in Bug 1950408 that resulted in this article [a] and this doc change [b]
Since this issues starts to happen only after the undercloud installation is completed, we can presume that some openstack containers are having bad mounts [5]
In Bugzilla 1950408 comment 18 [c], it's recommended that, if we need to bind mount /dev, we should also add a mount type=devpts,destination=/dev/pts.
It appears like tripleo doesn't have the hability to pass mounttypes.
[a] https://access.redhat.com/solutions/6205892
[b] https://github.com/openshift/openshift-docs/pull/34464/commits/b137fa8a5c9f7c715c9c8f2e8264008b048f1b59
[c] https://bugzilla.redhat.com/show_bug.cgi?id=1950408#c18
Steps to Reproduce:
1. Run script [4]
Workaround:
rm /dev/ptmx
mknod -m 666 /dev/ptmx c 5 2
or
chmod 666 /dev/pts/ptmx
Additional info:
[1]
~~~
(undercloud) [stack@undercloud-0 ~]$ ls -tlra /dev/ptmx
lrwxrwxrwx. 1 root root 8 Nov 7 12:34 /dev/ptmx -> pts/ptmx
~~~
[2]
~~~
[dvd@fedora ~]$ ls -tlra /dev/ptmx
crw-rw-rw-. 1 root tty 5, 2 Nov 8 09:39 /dev/ptmx
[dvd@fedora ~]$ ls -tlra /dev/pts/ptmx
c---------. 1 root root 5, 2 Oct 25 16:50 /dev/pts/ptmx
~~~
[3]
~~~
14:48:20.542970 openat(AT_FDCWD, "/dev/ptmx", O_RDWR) = -1 EACCES (Permission denied)
14:48:20.543061 write(2, "Traceback (most recent call last):\n", 35Traceback (most recent call last):
) = 35
14:48:20.543116 write(2, " File \"/home/stack/./test.py\", line 10, in <module>\n", 53 File "/home/stack/./test.py", line 10, in <module>
) = 53
14:48:20.543168 openat(AT_FDCWD, "/home/stack/./test.py", O_RDONLY|O_CLOEXEC) = 3
14:48:20.543216 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=437, ...}, AT_EMPTY_PATH) = 0
14:48:20.543266 ioctl(3, TCGETS, 0x7ffe22bd1f30) = -1 ENOTTY (Inappropriate ioctl for device)
14:48:20.543310 lseek(3, 0, SEEK_CUR) = 0
14:48:20.543361 fcntl(3, F_DUPFD_CLOEXEC, 0) = 4
14:48:20.543403 fcntl(4, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
14:48:20.543445 newfstatat(4, "", {st_mode=S_IFREG|0755, st_size=437, ...}, AT_EMPTY_PATH) = 0
14:48:20.543493 read(4, "#!/usr/bin/env python\n# Python program to explain os.openpty() method \n \n# importing os module \nimport os\n \n \n# open new pseudo-terminal pair\n# using os.openpty() method\nmaster, slave = os.openpty()\n \n \n# Get the terminal device\n# name associated with\n# file descriptor master \nname = os.ttyname(master)\nprint(name)\n \n \n# Get the terminal device\n# name associated with\n# file descriptor slave\nname = os.ttyname(slave)\nprint(name)\n", 4096) = 437
14:48:20.543542 close(4) = 0
14:48:20.543584 lseek(3, 0, SEEK_SET) = 0
14:48:20.543636 read(3, "#!/usr/bin/env python\n# Python program to explain os.openpty() method \n \n# importing os module \nimport os\n \n \n# open new pseudo-terminal pair\n# using os.openpty() method\nmaster, slave = os.openpty()\n \n \n# Get the terminal device\n# name associated with\n# file descriptor master \nname = os.ttyname(master)\nprint(name)\n \n \n# Get the terminal device\n# name associated with\n# file descriptor slave\nname = os.ttyname(slave)\nprint(name)\n", 8192) = 437
14:48:20.543689 close(3) = 0
14:48:20.543735 write(2, " master, slave = os.openpty()\n", 33 master, slave = os.openpty()
) = 33
14:48:20.543794 write(2, "PermissionError: [Errno 13] Permission denied\n", 46PermissionError: [Errno 13] Permission denied
) = 46
~~~
[4]
~~~
#!/usr/bin/env python
import os
master, slave = os.openpty()
~~~
[5]
~~~
[root@undercloud-0 ~]# podman ps -q | while read l;do echo "$l";podman inspect $l | jq -r '.[].Mounts[] | [.Source, .Destination] | @tsv' | grep -P "/dev[^\/]";done
229b0319bb29
560fa91d2da3
82bccf849215
fe0cdd41acb8
/dev /dev/
c7c11279eacb
7f842c077598
c22873137bfe
2ad7c661e2c0
8a703f6f40f9
0546bb29d745
88df372ab2af
8100522a45d4
/dev /dev
d4725a57676b
9940f74ff548
63a170016303
6d978a2fc69a
0b7824b15fe9
[root@undercloud-0 ~]# podman ps | grep -P "fe0cdd41acb8|8100522a45d4"
fe0cdd41acb8 undercloud-0.ctlplane.home.arpa:8787/tripleo_centos9/openstack-iscsid:latest kolla_start 2 days ago Up 2 days ago (healthy) iscsid
8100522a45d4 undercloud-0.ctlplane.home.arpa:8787/tripleo_centos9/openstack-ironic-conductor:latest kolla_start 2 days ago Up 2 days ago (unhealthy) ironic_conductor
~~~
I think We are also seeing similar issue on CS9 multinode jobs
https:/
```
/ansible/
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 INFO tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 INFO tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.