^x ^c while running an editor inside do-release-upgrade's screen session disconnects part of the session

Bug #1343737 reported by Peter Cordes
26
This bug affects 6 people
Affects Status Importance Assigned to Milestone
ubuntu-release-upgrader (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

on my seldom-used laptop, (running 12.04LTS last updates early august 2013) I ran
sudo do-release-upgrade -p
from bash in a gnome-terminal.

 It proceeded fine, until for one of the changed-config-file prompts I had it start a shell to examine the situation. I ran jed, a small emacslike editor, but immediately after I hit ^x ^c to exit jed, the screen session started by do-release-upgrade put up the message that the process had exitted, and did I want to close the screen window or restart the command. Attempted restarts gave the message that the dpkg database was already locked. From another shell, it turned out that jed was actually still running.

 I suspect I might have hit the ^x ^c, or maybe the preceding ^x ^s, following a ^space. do-release-upgrade starts screen with the escape character ^space, using the -e option. So to mark a region in an emacslike editor, I needed to get screen to pass through a ^space literally, so I had to press ^space space. I maybe have fumbled this and been in the middle of a screen escape sequence when trying to exit jed. Although neither ^x nor ^c, or even ^s (^x ^s = save in emacslike editors) should have done anything like what I observed, as screen escape commands.

 I unfortunately don't still have the situation fully available to poke at, as running firefox on the laptop completely hosed the system. Something must have tried to composite something, but the combination of updated and still running libdrm and nouveau libraries and kernel module just completely did not get along. After a big pause, I was able to switch to a text console, but eventually I just sshed in from my desktop to run ubuntu-bug. Anyway, I no longer have the whole process tree to look at or post, so I'm just working from memory.

 The gnome-session - bash - sudo - do-release-upgrade - screen - screen - dpkg - bash - jed
process tree was still there, but I think even at that point, the trusty - dpkg process tree had split off and was a child of init (pid 1).

 I've attached the pstree output from the system after exitting X, which killed all the do-release-upgrade stuff. and here's some poking with ps and ls /proc/pid/fd, from an ssh session:

# ps auxw | egrep 'dpkg|trusty'
root 6402 0.0 0.2 22748 9920 pts/1 Ss+ 15:15 0:04 /usr/bin/dpkg --force-overwrite --status-fd 75 --configure ttf-dejavu-core:all libcamel-1.2-45:amd64 libgcr-ui-3-1:amd64 libgcr-3-1:amd64 dconf-service:amd64 dconf-gsettings-backend:amd64 libcap-ng0:amd64 gcr:amd64 p11-kit-modules:amd64 p11-kit-modules:i386 p11-kit:amd64 libcap2-bin:amd64 gnome-keyring:amd64 libgnome-keyring-common:all libgnome-keyring0:amd64 libgirepository-1.0-1:amd64 gir1.2-glib-2.0:amd64 gir1.2-gnomekeyring-1.0:amd64 libsecret-common:all libsecret-1-0:amd64 libavahi-glib1:amd64 libassuan0:amd64 libgpgme11:amd64 seahorse:amd64 evolution-data-server-common:all libedataserver-1.2-18:amd64 libebook-contacts-1.2-0:amd64 libebackend-1.2-7:amd64 ... prob irrelevant so cutting out the clutter.
root 11781 0.0 0.0 9448 872 pts/11 S+ 23:22 0:00 egrep dpkg|trusty
root 12306 0.8 2.6 288968 106104 ? S 14:32 4:24 /usr/bin/python /tmp/update-manager-Y4hcGh/trusty --mode=server --frontend=DistUpgradeViewText

root@abacus:~# ll /proc/12306/fd
total 0
lrwx------ 1 root root 64 Jul 17 23:17 0 -> /dev/pts/4 (deleted)
lrwx------ 1 root root 64 Jul 17 23:17 1 -> /dev/pts/4 (deleted)
lr-x------ 1 root root 64 Jul 17 23:17 10 -> /var/lib/apt/lists/archive.canonical.com_ubuntu_dists_trusty_partner_binary-amd64_Packages
lr-x------ 1 root root 64 Jul 17 23:17 11 -> /var/lib/apt/lists/mirror.its.dal.ca_ubuntu_dists_trusty-updates_universe_i18n_Translation-en
 (edit: trimmed out the open file descriptors for more package lists and translations, beween 11 and 70)
lrwx------ 1 root root 64 Jul 17 23:17 2 -> /dev/pts/4 (deleted)
l-wx------ 1 root root 64 Jul 17 23:17 3 -> /var/log/dist-upgrade/20140717-lr-x------ 1 root root 64 Jul 17 23:17 36 -> /var/lib/apt/lists/mirror.its.dal.ca_ubuntu_dists_trusty_main_binary-amd64_Packages
lrwx------ 1 root root 64 Jul 17 23:17 37 -> /var/log/dist-upgrade/20140717-1529/apt.log
lrwx------ 1 root root 64 Jul 17 23:17 38 -> /dev/pts/4 (deleted)
lrwx------ 1 root root 64 Jul 17 23:17 39 -> /var/log/dist-upgrade/20140717-1529/apt.log
lr-x------ 1 root root 64 Jul 17 23:17 4 -> pipe:[161795]
lr-x------ 1 root root 64 Jul 17 23:17 40 -> /var/lib/dpkg/status (deleted)
l-wx------ 1 root root 64 Jul 17 23:17 5 -> pipe:[161795]
l-wx------ 1 root root 64 Jul 17 23:17 6 -> /var/log/dist-upgrade/20140717-1529/apt-term.log
lr-x------ 1 root root 64 Jul 17 23:17 69 -> /var/lib/apt/lists/mirror.its.dal.ca_ubuntu_dists_precise_restricted_binary-amd64_Packages
lrwx------ 1 root root 64 Jul 17 23:17 7 -> /var/lib/apt/lists/lock
lr-x------ 1 root root 64 Jul 17 23:17 70 -> /var/lib/apt/lists/mirror.its.dal.ca_ubuntu_dists_precise_main_binary-amd64_Packages
lrwx------ 1 root root 64 Jul 17 23:17 71 -> /var/log/dist-upgrade/20140717-1529/apt.log
lrwx------ 1 root root 64 Jul 17 23:17 72 -> /dev/pts/4 (deleted)
l-wx------ 1 root root 64 Jul 17 23:17 73 -> /var/log/dist-upgrade/20140717-1529/history.log
lr-x------ 1 root root 64 Jul 17 23:17 74 -> pipe:[618808]
lrwx------ 1 root root 64 Jul 17 23:17 76 -> /dev/ptmx
lr-x------ 1 root root 64 Jul 17 23:17 8 -> /var/lib/dpkg/status (deleted)
lr-x------ 1 root root 64 Jul 17 23:17 9 -> /var/lib/apt/lists/archive.canonical.com_ubuntu_dists_trusty_partner_binary-i386_Packages

# and the dpkg process
root@abacus:~# ll /proc/6402/fd
total 0
lrwx------ 1 root root 64 Jul 17 15:47 0 -> /dev/pts/1
lrwx------ 1 root root 64 Jul 17 15:47 1 -> /dev/pts/1
lrwx------ 1 root root 64 Jul 17 15:47 2 -> /dev/pts/1
lrwx------ 1 root root 64 Jul 17 15:47 3 -> /var/lib/dpkg/lock
l-wx------ 1 root root 64 Jul 17 15:47 4 -> /var/lib/dpkg/updates/tmp.i
lrwx------ 1 root root 64 Jul 17 15:47 5 -> /var/lib/dpkg/triggers/Lock
l-wx------ 1 root root 64 Jul 17 15:47 6 -> /var/log/dpkg.log
lr-x------ 1 root root 64 Jul 17 15:47 7 -> /var/lib/dpkg/diversions
lrwx------ 1 root root 64 Jul 17 15:47 71 -> /var/log/dist-upgrade/20140717-1529/apt.log
lrwx------ 1 root root 64 Jul 17 15:47 72 -> /dev/pts/4 (deleted)
l-wx------ 1 root root 64 Jul 17 15:47 75 -> pipe:[618808]

so the most interesting thing is that the trusty script, and dpkg, are somehow daemonized. They didn't exit when their parent terminal closed. And /dev/pts/1 isn't even deleted, even though the screen session is dead. I believe that the gnome-terminal tab I started sudo do-release-upgrade from may have been using the pts/1 pty.

root@abacus:~# ll /dev/pts/[0-5]
crw--w---- 1 lightdm tty 136, 0 Jul 17 16:45 /dev/pts/0
crw--w---- 1 root tty 136, 1 Jul 17 15:46 /dev/pts/1
crw--w---- 1 lightdm tty 136, 2 Jul 17 16:00 /dev/pts/2
crw--w---- 1 lightdm tty 136, 3 Jul 17 16:17 /dev/pts/3
crw--w---- 1 root tty 136, 5 Jul 17 15:22 /dev/pts/5 # my ssh session

 Anyway, now that I have this reported, I'll see if I can get my laptop cleaned up and working :P I foresee some happy hours with aptitude. It was fairly far along in the upgrade, to the point where lsb_release -rd shows
Description: Ubuntu 14.04 LTS
Release: 14.04
for the record, since that's in the bug report guidelines.

 Also, it's pretty cool that do-release-upgrade uses screen. I used ^s (xoff flow control), then ^space [ to pause and scroll back to look at stuff that caught my eye while scrolling past. I normally run everything from a screen session in a gnome-terminal tab managed by fluxbox, and have done so for about 10 years. So I'm very used to using screen. Never felt the need for a graphical shell, just a window manager to let me alt tab between my terminal and GUI programs.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: ubuntu-release-upgrader-core 1:0.220.2
ProcVersionSignature: Ubuntu 3.8.0-28.41~precise1-generic 3.8.13.5
Uname: Linux 3.8.0-28-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
CrashDB: ubuntu
Date: Thu Jul 17 19:46:00 2014
EcryptfsInUse: Yes
InstallationDate: Installed on 2009-12-14 (1676 days ago)
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
PackageArchitecture: all
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_CA.UTF-8
 SHELL=/bin/bash
SourcePackage: ubuntu-release-upgrader
Symptom: ubuntu-release-upgrader
UpgradeStatus: Upgraded to trusty on 2014-07-17 (0 days ago)

Revision history for this message
Peter Cordes (peter-cordes) wrote :
description: updated
Revision history for this message
Peter Cordes (peter-cordes) wrote :

Oh! I apparently missed a line in the Screenlog when I was looking before. In the Screenlog that ubuntu-bug gathered up, right after the last of jed's stuff, there's a
=== Command terminated with signal 2 (Thu Jul 17 15:29:19 2014) ===

(at the very end, before the two restart attempts)

 Which presumably came from the ^c as part of ^x ^c to exit jed. That was my other theory on what had happened. And that would explain how the trusty script got daemonized: its parent exitted, so it became a child of init, and no longer got sighup when my gnome-terminal died.

 The "mainlog.txt" is useless, it's from the 2nd attempt to restart. If you want it, I can upload the files from /var/log/dist-upgrade/20140717-1529/
-rw-r--r-- 1 root root 191K Jul 17 14:25 apt.log
-rw-r----- 1 root adm 945K Jul 17 15:46 apt-term.log
-rw-r--r-- 1 root root 141K Jul 17 14:33 history.log
-rw-r--r-- 1 root root 53K Jul 17 14:32 main.log

 But your best bet is to try using ^c while in the shell spawned from an "examine the situation" conf files prompt, or an editor from that. (as I said, I used jed, in case it does something to the terminal that's slightly different from what emacs or vim would do). If that produces similar results, there's your culprit. I'm not sure where would be the best place to try to fix it, if that's the case. Maybe some kind of terminal settings? But how to do that without turning off ctrl interrupts ALL the time, not just when there's fullscreen text program running from a subshell.

summary: - weird problem running an editor inside do-release-upgrade's screen
- session
+ ^x ^c while running an editor inside do-release-upgrade's screen session
+ disconnects part of the session
Revision history for this message
Peter Cordes (peter-cordes) wrote :

killing the running dpkg and then trusty processes changed the timestamp on the files in /var/log/dist-upgrade/20140717-1529/apt-term.log and history.log, so they were indeed still open.

 dpkg --configure -a finished ok, so looks like my laptop will be fine. yay for robust package systems.

 The end of apt-term.log is
.... bunch of colored text that's missing jed's cursor-movement stuff when I view it with less ...

                                                               ^M***Fatal Error: Killed by signal 15.

jed version: 0.99.19/Unix
 Compiled with GNU C 4.6
S-Lang version: 2.2.4

jed compile-time options:
 +LINE_ATTRIBUTES +BUFFER_LOCAL_VARS +SAVE_NARROW +TTY_MENUS
 +EMACS_LOCKING +MULTICLICK +SUBPROCESSES +DFA_SYNTAX +ABBREVS
 +COLOR_COLUMNS +LINE_MARKS +GPM_MOUSE +IMPORT
CBuf: 0x7f9c5c5f4910, CLine: 0x7f9c5c590630, Point 0
CLine: data: 0x7f9c5c5cc7f0, len = 18, next: 0x7f9c5c590650, prev (nil)
Max_LineNum: 39, LineNum: 1
JWindow: 0x7f9c5c5ca9f0, top: 1, rows: 40, buffer: 0x7f9c5c5f4910
root@abacus:/etc/etckeeper# exit

Configuration file '/etc/etckeeper/etckeeper.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ? Your options are:
    Y or I : install the package maintainer's version
    N or O : keep your currently-installed version
      D : show the differences between the versions
      Z : start a shell to examine the situation
 The default action is to keep your current version.
*** etckeeper.conf (Y/I/N/O/D/Z) [default=N] ? Log ended: 2014-07-17 23:51:45
-----

All that's new is the Log ended: stuff. The rest was all there even before I tried to start firefox and ended up exitting X. So it had re-prompted for what to do about the conffile even after jed got a sigterm (which I sent with kill from another shell). The "exit" on the bash prompt was printed by bash when I sent it a sigterm.

  (btw, stop lightdm finally fixed the intermittent pauses that were affecting even the ssh session, it must have been starting new X servers every couple minutes. Totally unconnected to this bug, a symptom of where the upgrade got stuck. Just feeling chatty I guess.)

 Anyway, note that apt-term.log did NOT include the signal 2 (sigint), so it wasn't a process running in that terminal that got the sigint from the ^c, it was somewhere above the parent of the trusty upgrade script, I guess. I hadn't looked at screenlog, only apt-term.log, until I was done typing the original bug report.

 I changed the bug title to match the theory that it was the ^c that caused the problem, not a screen escape sequence.

 Also meant to mention that this is slightly similar to
https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1302218
where the submitter tried to start a subshell from inside the terminal widget in the gui release-upgrader.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu-release-upgrader (Ubuntu):
status: New → Confirmed
Revision history for this message
Joachim Durchholz (jo-durchholz) wrote :

I just had the same problem: I wanted to inspect a merge conflict in conffiles, went into a shell and started joe, pressed Ctrl-C to exit joe without changes, just to see the session disappear.
Taskmanager confirmed that joe was still running in the background.

Workarounds:

FAILED: screen wouldn't work because I did not have it installed, and I couldn't do apt-get because the package repository was locked.

FAILED: I was told by a program (apt-get probably but I'm not sure) told me that a reboot might help.
Fortunately, that didn't work either: the reboot command said something about unfinished business (the message sounded like systemd terminology but I can't say for sure, and I don't want to retry with a half-upgraded system: the reboot might work, and I don't want that anymore).

WORKED: I found, downloaded and compiled reptyr ("re-pty-er").
Running "reptyr -T <pid of the editor process>" reconnected me. I haven't closed the editor yet (I'll just kill it if I find I can't close it without pressing Ctrl-C).
This worked because I happened to have git, gcc and related buildtools installed; I'm extremely lucky because I had considered uninstalling gcc because it's been more than a year since I last touched it; I don't know how to install a C compiler without a package manager (now *that* would have been fun...)

CONCLUSION:
I find it pretty harsh to put people in an environment where a single keypress can put them so deeply into such a tricky-to-escape-from situation.
I'm still pretty anxious because I don't know whether I can return to the entire process when I close the editor - the -T option hopefully has taken care of that, but reptyr is doing pretty bizarre and arcane stuff, which means that its failure modes will be even more bizarre and arcane. Hopefully I won't encounter any of these failure modes, but it's a scary situation to be in!

Revision history for this message
Joachim Durchholz (jo-durchholz) wrote :

More background information:

I am upgrading from bionic to cosmic.

The process hierarchy is, starting with PID 1 and following the chain down to the editor process:

/lib/systemd/systemd --system --deserialize 22
/usr/bin/python3 /tmp/ubuntu-release-upgrader-11yhdo4c/cosmic --mode=server --frontend=DistUpgradeViewText
/usr/bin/dpkg --force-overwrite --statud-fd 122 --configure --pending
/usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/unattended-upgrades.postinst configure 1.1ubuntu1.18.04.11
/bin/sh /&var/lib/dpkg/info/unattended-upgrades.postinst configure 1.1ubuntu1.18.04.11
/bin/bash /usr/bin/ucf --three-way --debconf-ok /etc/apt/apd.conf.d/50unattended-uptrades.ucftmp /etc/apt/apt.conf.d/50unattended-upgrades
bash
joe 50unattended-upgrades 50unattended-upgrades~ 50unattended-upgrades.merge-error 50unattended-upgrades.ucf-dist 50unattended-upgrades.ucftmp

I had to type the commands manually, so typos are possible (and likely).

Continuing... fingers crossed...

Revision history for this message
Joachim Durchholz (jo-durchholz) wrote :

Hmm... do-release-upgrade seems to have finished, but reptyer does not terminate.
Killing it, rebooting, crossing fingers...

Revision history for this message
Joachim Durchholz (jo-durchholz) wrote :

... phew. I had to restart do-release-upgrade, but in the end, everything would work.

Revision history for this message
Nathan Collins (ntc2) wrote :

I just had the same problem: I chose "Z" to drop into a shell to examine a conf file upgrade conflict. Used Emacs to edit the conf file and then pressed ^x^c to exit and immediately had trouble.

I'm now waiting for dpkg --configure -a to finish and hoping for the best ...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.