| 2020-04-01 09:25:07 |
Mark Shuttleworth |
bug |
|
|
added bug |
| 2020-04-01 19:59:57 |
Simon Poirier |
landscape-client: status |
New |
Triaged |
|
| 2020-04-01 20:00:09 |
Simon Poirier |
landscape-client: importance |
Undecided |
High |
|
| 2020-04-01 20:00:58 |
Daniel Manrique |
landscape-client: assignee |
|
Simon Poirier (simpoir) |
|
| 2020-04-06 15:04:30 |
Simon Poirier |
landscape-client: status |
Triaged |
In Progress |
|
| 2020-04-06 15:43:22 |
Bret Barker |
bug |
|
|
added subscriber Bret Barker |
| 2020-04-06 22:34:33 |
Simon Poirier |
bug watch added |
|
http://twistedmatrix.com/trac/ticket/9738 |
|
| 2020-04-10 15:01:06 |
Simon Poirier |
landscape-client: status |
In Progress |
Fix Committed |
|
| 2020-04-14 16:03:24 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~simpoir/ubuntu/+source/landscape-client/+git/landscape-client/+merge/382235 |
|
| 2020-04-14 19:51:43 |
Eric Desrochers |
bug |
|
|
added subscriber STS Sponsors |
| 2020-04-14 19:51:51 |
Eric Desrochers |
tags |
|
sts-sponsor-slashd |
|
| 2020-04-14 19:55:11 |
Simon Poirier |
bug task added |
|
landscape-client (Ubuntu) |
|
| 2020-04-14 19:55:26 |
Simon Poirier |
nominated for series |
|
Ubuntu Focal |
|
| 2020-04-14 19:55:26 |
Simon Poirier |
bug task added |
|
landscape-client (Ubuntu Focal) |
|
| 2020-04-14 19:56:11 |
Simon Poirier |
landscape-client (Ubuntu Focal): status |
New |
In Progress |
|
| 2020-04-14 19:56:15 |
Simon Poirier |
landscape-client (Ubuntu Focal): assignee |
|
Simon Poirier (simpoir) |
|
| 2020-04-14 20:08:47 |
Eric Desrochers |
landscape-client (Ubuntu Focal): importance |
Undecided |
High |
|
| 2020-04-14 20:15:34 |
Simon Poirier |
description |
I have a machine which was failing to connect to the landscape service. In syslog I found this traceback:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 160, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: symlink(str(os.getpid()), self.name)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: FileExistsError: [Errno 17] File exists: '1538397' -> b'/var/lib/landscape/client/sockets/broker.sock.lock'
Apr 1 03:27:53 maas-1 landscape-client[1538354]: During handling of the above exception, another exception occurred:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/bin/landscape-broker", line 8, in <module>
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run(sys.argv)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 93, in run
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run_landscape_service(BrokerConfiguration, BrokerService, args)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/service.py", line 115, in run_landscape_service
Apr 1 03:27:53 maas-1 landscape-client[1538354]: startApplication(application, False)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 690, in startApplication
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.IService(application).startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 288, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 79, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self.publisher.start()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/amp.py", line 45, in start
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self._port = self._reactor.listen_unix(socket_path, factory)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/lib/reactor.py", line 228, in listen_unix
Apr 1 03:27:53 maas-1 landscape-client[1538354]: return self._reactor.listenUNIX(socket, factory, wantPID=True)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 397, in listenUNIX
Apr 1 03:27:53 maas-1 landscape-client[1538354]: p.startListening()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/unix.py", line 372, in startListening
Apr 1 03:27:53 maas-1 landscape-client[1538354]: if not self.lockFile.lock():
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 185, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: kill(int(pid), 0)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: PermissionError: [Errno 1] Operation not permitted
In the sockets directory I saw:
$ sudo ls /var/lib/landscape/client/sockets/ -la
total 8
drwxr-x--- 2 landscape root 4096 Apr 1 03:27 .
drwxr-xr-x 7 landscape root 4096 Apr 1 03:27 ..
srw-rw-rw- 1 landscape landscape 0 Mar 12 01:41 broker.sock
lrwxrwxrwx 1 landscape landscape 3 Mar 12 01:41 broker.sock.lock -> 905
Removing those two files allowed the landscape client to start as normal.
Looks like we need some lockfile cleanup code on start. |
[Impact]
* landscape-client services are prevented from starting if its older PIDs get
recycled.
* The exact conditions for the issue, are particularly more likely to occur
on release upgrade.
* The proposed fix tries to verify existing locks actually belong
to landscape-client, instead of just verifying they exist.
[Test Case]
* systemctl stop landscape-client
* ln -sf 1 /var/lib/landscape/client/sockets/broker.sock.lock
* systemctl start landscape-client
[Regression Potential]
* The existing twisted logic is still kept, so assuming checking process
names fail, lock conflicts should still be detected normally.
* The locks which twisted creates are unlikely to actually see conflicts in
the wild as those processes are managed by systemd. False positives in
the detection check should have minimal impact.
[Original description]
I have a machine which was failing to connect to the landscape service. In syslog I found this traceback:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 160, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: symlink(str(os.getpid()), self.name)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: FileExistsError: [Errno 17] File exists: '1538397' -> b'/var/lib/landscape/client/sockets/broker.sock.lock'
Apr 1 03:27:53 maas-1 landscape-client[1538354]: During handling of the above exception, another exception occurred:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/bin/landscape-broker", line 8, in <module>
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run(sys.argv)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 93, in run
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run_landscape_service(BrokerConfiguration, BrokerService, args)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/service.py", line 115, in run_landscape_service
Apr 1 03:27:53 maas-1 landscape-client[1538354]: startApplication(application, False)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 690, in startApplication
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.IService(application).startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 288, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 79, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self.publisher.start()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/amp.py", line 45, in start
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self._port = self._reactor.listen_unix(socket_path, factory)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/lib/reactor.py", line 228, in listen_unix
Apr 1 03:27:53 maas-1 landscape-client[1538354]: return self._reactor.listenUNIX(socket, factory, wantPID=True)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 397, in listenUNIX
Apr 1 03:27:53 maas-1 landscape-client[1538354]: p.startListening()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/unix.py", line 372, in startListening
Apr 1 03:27:53 maas-1 landscape-client[1538354]: if not self.lockFile.lock():
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 185, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: kill(int(pid), 0)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: PermissionError: [Errno 1] Operation not permitted
In the sockets directory I saw:
$ sudo ls /var/lib/landscape/client/sockets/ -la
total 8
drwxr-x--- 2 landscape root 4096 Apr 1 03:27 .
drwxr-xr-x 7 landscape root 4096 Apr 1 03:27 ..
srw-rw-rw- 1 landscape landscape 0 Mar 12 01:41 broker.sock
lrwxrwxrwx 1 landscape landscape 3 Mar 12 01:41 broker.sock.lock -> 905
Removing those two files allowed the landscape client to start as normal.
Looks like we need some lockfile cleanup code on start. |
|
| 2020-04-14 20:35:37 |
Eric Desrochers |
removed subscriber STS Sponsors |
|
|
|
| 2020-04-14 20:35:40 |
Eric Desrochers |
bug |
|
|
added subscriber Eric Desrochers |
| 2020-04-14 20:35:46 |
Eric Desrochers |
tags |
sts-sponsor-slashd |
|
|
| 2020-04-15 15:09:53 |
Launchpad Janitor |
landscape-client (Ubuntu Focal): status |
In Progress |
Fix Released |
|
| 2020-04-21 22:23:49 |
Simon Poirier |
merge proposal linked |
|
https://code.launchpad.net/~simpoir/ubuntu/+source/landscape-client/+git/landscape-client/+merge/382682 |
|
| 2020-04-22 15:55:32 |
Daniel Manrique |
landscape-client (Ubuntu Focal): status |
Fix Released |
In Progress |
|
| 2020-04-22 15:55:35 |
Daniel Manrique |
landscape-client: status |
Fix Committed |
In Progress |
|
| 2020-04-24 17:27:01 |
Eric Desrochers |
bug |
|
|
added subscriber STS Sponsors |
| 2020-04-24 17:27:09 |
Eric Desrochers |
tags |
|
sts-sponsor-slashd |
|
| 2020-04-27 15:36:11 |
Eric Desrochers |
nominated for series |
|
Ubuntu Groovy |
|
| 2020-04-27 15:36:11 |
Eric Desrochers |
bug task added |
|
landscape-client (Ubuntu Groovy) |
|
| 2020-04-27 16:20:23 |
Eric Desrochers |
attachment added |
|
groovy.debdiff https://bugs.launchpad.net/landscape-client/+bug/1870087/+attachment/5361756/+files/groovy.debdiff |
|
| 2020-04-27 16:29:00 |
Ubuntu Foundations Team Bug Bot |
tags |
sts-sponsor-slashd |
patch sts-sponsor-slashd |
|
| 2020-04-27 17:27:07 |
Launchpad Janitor |
landscape-client (Ubuntu Groovy): status |
In Progress |
Fix Released |
|
| 2020-04-27 20:56:03 |
Łukasz Zemczak |
landscape-client (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2020-04-27 20:56:05 |
Łukasz Zemczak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2020-04-27 20:56:08 |
Łukasz Zemczak |
bug |
|
|
added subscriber SRU Verification |
| 2020-04-27 20:56:12 |
Łukasz Zemczak |
tags |
patch sts-sponsor-slashd |
patch sts-sponsor-slashd verification-needed verification-needed-focal |
|
| 2020-04-30 15:30:44 |
Eric Desrochers |
removed subscriber STS Sponsors |
|
|
|
| 2020-05-04 15:41:55 |
Simon Poirier |
tags |
patch sts-sponsor-slashd verification-needed verification-needed-focal |
patch sts-sponsor-slashd verification-done verification-done-focal |
|
| 2020-05-05 20:05:28 |
Launchpad Janitor |
landscape-client (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2020-05-05 20:05:32 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2020-08-03 13:50:50 |
Simon Poirier |
landscape-client (Ubuntu Focal): status |
Fix Released |
In Progress |
|
| 2020-08-27 22:28:33 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~simpoir/ubuntu/+source/landscape-client/+git/landscape-client/+merge/389933 |
|
| 2020-08-27 22:39:26 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~simpoir/ubuntu/+source/landscape-client/+git/landscape-client/+merge/389935 |
|
| 2020-08-28 21:33:51 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~simpoir/ubuntu/+source/landscape-client/+git/landscape-client/+merge/389989 |
|
| 2020-08-28 22:31:08 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~simpoir/ubuntu/+source/landscape-client/+git/landscape-client/+merge/389991 |
|
| 2021-03-02 16:54:50 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~simpoir/ubuntu/+source/landscape-client/+git/landscape-client/+merge/398996 |
|
| 2021-03-02 17:01:58 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~simpoir/ubuntu/+source/landscape-client/+git/landscape-client/+merge/399000 |
|
| 2021-03-02 17:05:45 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~simpoir/ubuntu/+source/landscape-client/+git/landscape-client/+merge/399001 |
|
| 2021-03-11 15:23:58 |
Eric Desrochers |
bug |
|
|
added subscriber STS Sponsors |
| 2021-03-11 16:30:42 |
Chuck Short |
bug |
|
|
added subscriber Chuck Short |
| 2021-03-11 17:33:42 |
Simon Poirier |
description |
[Impact]
* landscape-client services are prevented from starting if its older PIDs get
recycled.
* The exact conditions for the issue, are particularly more likely to occur
on release upgrade.
* The proposed fix tries to verify existing locks actually belong
to landscape-client, instead of just verifying they exist.
[Test Case]
* systemctl stop landscape-client
* ln -sf 1 /var/lib/landscape/client/sockets/broker.sock.lock
* systemctl start landscape-client
[Regression Potential]
* The existing twisted logic is still kept, so assuming checking process
names fail, lock conflicts should still be detected normally.
* The locks which twisted creates are unlikely to actually see conflicts in
the wild as those processes are managed by systemd. False positives in
the detection check should have minimal impact.
[Original description]
I have a machine which was failing to connect to the landscape service. In syslog I found this traceback:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 160, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: symlink(str(os.getpid()), self.name)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: FileExistsError: [Errno 17] File exists: '1538397' -> b'/var/lib/landscape/client/sockets/broker.sock.lock'
Apr 1 03:27:53 maas-1 landscape-client[1538354]: During handling of the above exception, another exception occurred:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/bin/landscape-broker", line 8, in <module>
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run(sys.argv)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 93, in run
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run_landscape_service(BrokerConfiguration, BrokerService, args)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/service.py", line 115, in run_landscape_service
Apr 1 03:27:53 maas-1 landscape-client[1538354]: startApplication(application, False)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 690, in startApplication
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.IService(application).startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 288, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 79, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self.publisher.start()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/amp.py", line 45, in start
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self._port = self._reactor.listen_unix(socket_path, factory)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/lib/reactor.py", line 228, in listen_unix
Apr 1 03:27:53 maas-1 landscape-client[1538354]: return self._reactor.listenUNIX(socket, factory, wantPID=True)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 397, in listenUNIX
Apr 1 03:27:53 maas-1 landscape-client[1538354]: p.startListening()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/unix.py", line 372, in startListening
Apr 1 03:27:53 maas-1 landscape-client[1538354]: if not self.lockFile.lock():
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 185, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: kill(int(pid), 0)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: PermissionError: [Errno 1] Operation not permitted
In the sockets directory I saw:
$ sudo ls /var/lib/landscape/client/sockets/ -la
total 8
drwxr-x--- 2 landscape root 4096 Apr 1 03:27 .
drwxr-xr-x 7 landscape root 4096 Apr 1 03:27 ..
srw-rw-rw- 1 landscape landscape 0 Mar 12 01:41 broker.sock
lrwxrwxrwx 1 landscape landscape 3 Mar 12 01:41 broker.sock.lock -> 905
Removing those two files allowed the landscape client to start as normal.
Looks like we need some lockfile cleanup code on start. |
[Impact]
* landscape-client services are prevented from starting if its older PIDs get
recycled.
* The exact conditions for the issue, are particularly more likely to occur
on release upgrade.
* The proposed fix tries to verify existing locks actually belong
to landscape-client, instead of just verifying they exist.
[Test Case]
* systemctl stop landscape-client
* There should not be any remaining file in /var/lib/landscape/client/sockets/
* ln -sf 1 /var/lib/landscape/client/sockets/broker.sock.lock
* systemctl start landscape-client
[Regression Potential]
* The existing twisted logic is still kept, so assuming checking process
names fail, lock conflicts should still be detected normally.
* The locks which twisted creates are unlikely to actually see conflicts in
the wild as those processes are managed by systemd. False positives in
the detection check should have minimal impact.
[Original description]
I have a machine which was failing to connect to the landscape service. In syslog I found this traceback:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 160, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: symlink(str(os.getpid()), self.name)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: FileExistsError: [Errno 17] File exists: '1538397' -> b'/var/lib/landscape/client/sockets/broker.sock.lock'
Apr 1 03:27:53 maas-1 landscape-client[1538354]: During handling of the above exception, another exception occurred:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/bin/landscape-broker", line 8, in <module>
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run(sys.argv)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 93, in run
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run_landscape_service(BrokerConfiguration, BrokerService, args)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/service.py", line 115, in run_landscape_service
Apr 1 03:27:53 maas-1 landscape-client[1538354]: startApplication(application, False)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 690, in startApplication
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.IService(application).startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 288, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 79, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self.publisher.start()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/amp.py", line 45, in start
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self._port = self._reactor.listen_unix(socket_path, factory)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/lib/reactor.py", line 228, in listen_unix
Apr 1 03:27:53 maas-1 landscape-client[1538354]: return self._reactor.listenUNIX(socket, factory, wantPID=True)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 397, in listenUNIX
Apr 1 03:27:53 maas-1 landscape-client[1538354]: p.startListening()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/unix.py", line 372, in startListening
Apr 1 03:27:53 maas-1 landscape-client[1538354]: if not self.lockFile.lock():
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 185, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: kill(int(pid), 0)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: PermissionError: [Errno 1] Operation not permitted
In the sockets directory I saw:
$ sudo ls /var/lib/landscape/client/sockets/ -la
total 8
drwxr-x--- 2 landscape root 4096 Apr 1 03:27 .
drwxr-xr-x 7 landscape root 4096 Apr 1 03:27 ..
srw-rw-rw- 1 landscape landscape 0 Mar 12 01:41 broker.sock
lrwxrwxrwx 1 landscape landscape 3 Mar 12 01:41 broker.sock.lock -> 905
Removing those two files allowed the landscape client to start as normal.
Looks like we need some lockfile cleanup code on start. |
|
| 2021-03-11 17:40:26 |
Simon Poirier |
description |
[Impact]
* landscape-client services are prevented from starting if its older PIDs get
recycled.
* The exact conditions for the issue, are particularly more likely to occur
on release upgrade.
* The proposed fix tries to verify existing locks actually belong
to landscape-client, instead of just verifying they exist.
[Test Case]
* systemctl stop landscape-client
* There should not be any remaining file in /var/lib/landscape/client/sockets/
* ln -sf 1 /var/lib/landscape/client/sockets/broker.sock.lock
* systemctl start landscape-client
[Regression Potential]
* The existing twisted logic is still kept, so assuming checking process
names fail, lock conflicts should still be detected normally.
* The locks which twisted creates are unlikely to actually see conflicts in
the wild as those processes are managed by systemd. False positives in
the detection check should have minimal impact.
[Original description]
I have a machine which was failing to connect to the landscape service. In syslog I found this traceback:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 160, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: symlink(str(os.getpid()), self.name)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: FileExistsError: [Errno 17] File exists: '1538397' -> b'/var/lib/landscape/client/sockets/broker.sock.lock'
Apr 1 03:27:53 maas-1 landscape-client[1538354]: During handling of the above exception, another exception occurred:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/bin/landscape-broker", line 8, in <module>
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run(sys.argv)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 93, in run
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run_landscape_service(BrokerConfiguration, BrokerService, args)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/service.py", line 115, in run_landscape_service
Apr 1 03:27:53 maas-1 landscape-client[1538354]: startApplication(application, False)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 690, in startApplication
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.IService(application).startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 288, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 79, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self.publisher.start()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/amp.py", line 45, in start
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self._port = self._reactor.listen_unix(socket_path, factory)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/lib/reactor.py", line 228, in listen_unix
Apr 1 03:27:53 maas-1 landscape-client[1538354]: return self._reactor.listenUNIX(socket, factory, wantPID=True)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 397, in listenUNIX
Apr 1 03:27:53 maas-1 landscape-client[1538354]: p.startListening()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/unix.py", line 372, in startListening
Apr 1 03:27:53 maas-1 landscape-client[1538354]: if not self.lockFile.lock():
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 185, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: kill(int(pid), 0)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: PermissionError: [Errno 1] Operation not permitted
In the sockets directory I saw:
$ sudo ls /var/lib/landscape/client/sockets/ -la
total 8
drwxr-x--- 2 landscape root 4096 Apr 1 03:27 .
drwxr-xr-x 7 landscape root 4096 Apr 1 03:27 ..
srw-rw-rw- 1 landscape landscape 0 Mar 12 01:41 broker.sock
lrwxrwxrwx 1 landscape landscape 3 Mar 12 01:41 broker.sock.lock -> 905
Removing those two files allowed the landscape client to start as normal.
Looks like we need some lockfile cleanup code on start. |
[Impact]
* landscape-client services are prevented from starting if its older PIDs get
recycled.
* The exact conditions for the issue, are particularly more likely to occur
on release upgrade. This is exacerbated by the fact clients did not await
on their shutdown routine, thus were likely to leak their lock file.
* The proposed fix tries to verify existing locks actually belong
to landscape-client, instead of just verifying they exist.
* The follow-up patch ensured some of the process actually complete their shutdown.
[Test Case]
* systemctl stop landscape-client
* There should not be any remaining file in /var/lib/landscape/client/sockets/
* ln -sf 1 /var/lib/landscape/client/sockets/broker.sock.lock
* systemctl start landscape-client
[Regression Potential]
* The existing twisted logic is still kept, so assuming checking process
names fail, lock conflicts should still be detected normally.
* The locks which twisted creates are unlikely to actually see conflicts in
the wild as those processes are managed by systemd. False positives in
the detection check should have minimal impact.
[Original description]
I have a machine which was failing to connect to the landscape service. In syslog I found this traceback:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 160, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: symlink(str(os.getpid()), self.name)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: FileExistsError: [Errno 17] File exists: '1538397' -> b'/var/lib/landscape/client/sockets/broker.sock.lock'
Apr 1 03:27:53 maas-1 landscape-client[1538354]: During handling of the above exception, another exception occurred:
Apr 1 03:27:53 maas-1 landscape-client[1538354]: Traceback (most recent call last):
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/bin/landscape-broker", line 8, in <module>
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run(sys.argv)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 93, in run
Apr 1 03:27:53 maas-1 landscape-client[1538354]: run_landscape_service(BrokerConfiguration, BrokerService, args)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/service.py", line 115, in run_landscape_service
Apr 1 03:27:53 maas-1 landscape-client[1538354]: startApplication(application, False)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 690, in startApplication
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.IService(application).startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 288, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: service.startService()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/broker/service.py", line 79, in startService
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self.publisher.start()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/client/amp.py", line 45, in start
Apr 1 03:27:53 maas-1 landscape-client[1538354]: self._port = self._reactor.listen_unix(socket_path, factory)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/landscape/lib/reactor.py", line 228, in listen_unix
Apr 1 03:27:53 maas-1 landscape-client[1538354]: return self._reactor.listenUNIX(socket, factory, wantPID=True)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 397, in listenUNIX
Apr 1 03:27:53 maas-1 landscape-client[1538354]: p.startListening()
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/internet/unix.py", line 372, in startListening
Apr 1 03:27:53 maas-1 landscape-client[1538354]: if not self.lockFile.lock():
Apr 1 03:27:53 maas-1 landscape-client[1538354]: File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 185, in lock
Apr 1 03:27:53 maas-1 landscape-client[1538354]: kill(int(pid), 0)
Apr 1 03:27:53 maas-1 landscape-client[1538354]: PermissionError: [Errno 1] Operation not permitted
In the sockets directory I saw:
$ sudo ls /var/lib/landscape/client/sockets/ -la
total 8
drwxr-x--- 2 landscape root 4096 Apr 1 03:27 .
drwxr-xr-x 7 landscape root 4096 Apr 1 03:27 ..
srw-rw-rw- 1 landscape landscape 0 Mar 12 01:41 broker.sock
lrwxrwxrwx 1 landscape landscape 3 Mar 12 01:41 broker.sock.lock -> 905
Removing those two files allowed the landscape client to start as normal.
Looks like we need some lockfile cleanup code on start. |
|
| 2021-03-11 17:41:39 |
Simon Poirier |
landscape-client (Ubuntu Groovy): status |
Fix Released |
In Progress |
|
| 2021-03-11 17:42:59 |
Chuck Short |
removed subscriber Chuck Short |
|
|
|
| 2021-03-23 19:16:00 |
Brian Murray |
landscape-client (Ubuntu Groovy): status |
In Progress |
Fix Committed |
|
| 2021-03-23 19:16:03 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2021-03-23 19:16:11 |
Brian Murray |
tags |
patch sts-sponsor-slashd verification-done verification-done-focal |
patch sts-sponsor-slashd verification-done-focal verification-needed verification-needed-groovy |
|
| 2021-03-23 19:18:51 |
Brian Murray |
landscape-client (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2021-03-23 19:19:00 |
Brian Murray |
tags |
patch sts-sponsor-slashd verification-done-focal verification-needed verification-needed-groovy |
patch sts-sponsor-slashd verification-needed verification-needed-focal verification-needed-groovy |
|
| 2021-04-02 17:06:05 |
Simon Poirier |
tags |
patch sts-sponsor-slashd verification-needed verification-needed-focal verification-needed-groovy |
patch sts-sponsor-slashd verification-done verification-done-focal verification-done-groovy |
|
| 2021-04-06 17:55:49 |
Launchpad Janitor |
landscape-client (Ubuntu Groovy): status |
Fix Committed |
Fix Released |
|
| 2021-04-06 17:57:36 |
Launchpad Janitor |
landscape-client (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-04-15 20:36:20 |
Eric Desrochers |
removed subscriber STS Sponsors |
|
|
|
| 2021-09-10 19:00:37 |
Simon Poirier |
landscape-client: status |
In Progress |
Fix Committed |
|
| 2021-09-10 19:00:48 |
Simon Poirier |
landscape-client: status |
Fix Committed |
Fix Released |
|
