A file with link count 2 doesn't get both links updated

This appears to be the problem I encountered after upgrading from devel r1 to devel r2. The full traceback follows:

Fatal error: /tmp/com.ubuntu.developer.rschroll.beru_1.0.0_armhf.click failed to install.
WARNING:root:Signature check failed, but installing anyway as requested
Traceback (most recent call last):
  File "/usr/bin/aa-clickhook", line 14, in <module>
    from apparmor import click
  File "/usr/lib/python3/dist-packages/apparmor/click.py", line 15, in <module>
    import ctypes
  File "/usr/lib/python3.4/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-arm-linux-gnueabihf.so: undefined symbol: _PyTraceback_Add
Traceback (most recent call last):
  File "/usr/bin/click", line 86, in <module>
    sys.exit(main())
  File "/usr/bin/click", line 82, in main
    return mod.run(args)
  File "/usr/lib/python3/dist-packages/click/commands/install.py", line 66, in run
    quiet=not options.verbose)
  File "/usr/lib/python3/dist-packages/click/install.py", line 457, in install
    path, user=user, all_users=all_users, quiet=quiet)
  File "/usr/lib/python3/dist-packages/click/install.py", line 441, in _unpack
    user_name=None)
GLib.Error: click_hooks_error-quark: Hook command '/usr/bin/aa-clickhook' failed: Child process exited with code 1 (6)

Status changed to 'Confirmed' because the bug affects multiple users.

AFAICT, this is not a problem in click-apparmor, but instead python3.4. aa-clickhook is importing /usr/lib/python3/dist-packages/apparmor/click.py and it finds it. /usr/lib/python3/dist-packages/apparmor/click.py imports 'ctypes' and it fails due to the missing symbol in /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-arm-linux-gnueabihf.so:

undefined symbol: _PyTraceback_Add

On up to date mako on devel-proposed, I cannot reproduce this:
$ sudo aa-clickhook -f # no error
$

$ nm -PoDC ./_ctypes.cpython-34m-arm-linux-gnueabihf.so |grep _PyTraceback_Add
./_ctypes.cpython-34m-arm-linux-gnueabihf.so: _PyTraceback_Add U

$ system-image-cli -i
current build number: 172
device name: mako
channel: ubuntu-touch/devel-proposed
alias: ubuntu-touch/vivid-proposed
last update: 2015-04-15 17:30:00
version version: 172
version ubuntu: 20150415
version device: 20150210
version custom: 20150415

Can someone with a device where this is reproducible please post the output of the following command, run against the /usr/bin/python3.4 image from the device?

  objdump -T /usr/bin/python3.4 | grep _PyTraceback_Add

objdump doesn't exist on the device, so I copied the python3.4 binary over to my desktop. Running that command doesn't produce any output. (Just objdump -T alone does, so I assume it's working but producing nothing that grep matches.)

FWIW, it looks like the only use for ctypes in click.py is to import the nih_dbus_path() function. But you can just write it in python in about as many lines as it took to import it:

def encode_byte(b):
    if ord(b'a') <= b <= ord(b'z') or ord(b'A') <= b <= ord(b'Z') or ord(b'0') <= b <= ord(b'9'):
        return bytes([b])
    return ('_%0.2x' % b).encode()

def my_dbus_path(x, *args):
    return b'/'.join(b''.join(encode_byte(b) for b in arg) for arg in args if arg is not None)

It probably doesn't fail in the same ways, but for a few tests with valid imports, it seems to work.

On my BQ Aquaris device (krillin) I am unable to reproduce the issue. When looking for the symbol in the python3.4 executable here I see that it's here:

objdump -T python3.4 | grep _PyTraceback_Add
0003b421 g DF .text 000000a8 Base _PyTraceback_Add

So it's *very* interesting to know that the binary on your device does not have this symbol. Both should be the same since they're installed from the very same armhf binary package. Strange!

Just so that there are no misunderstandings - I am running the ubuntu-touch/devel channel for that:

current build number: 2
device name: krillin
channel: devel
alias: ubuntu-touch/vivid
last update: 2015-04-15 22:33:51
version version: 2
version ubuntu: 20150413
version device: 20150326-f0c5ba5
version custom: 20150413

phablet@ubuntu-phablet:~$ system-image-cli -i
current build number: 2
device name: flo
channel: devel
alias: ubuntu-touch/vivid
last update: 2015-04-15 03:04:31
version version: 2
version ubuntu: 20150413
version device: 20150210
version custom: 20150413
phablet@ubuntu-phablet:~$ md5sum /usr/bin/python3.4
42cc689fa8d2332cefe133cd771fbe6b /usr/bin/python3.4

I upgraded this from r1. If you'd like, I can flash it anew and see if that changes things.

We've traced this down to a problem with the generation of the system-image delta at the time the image was promoted from the devel-proposed to the devel channel. The /usr/bin/python3.4 binary is missing from the delta, even though /usr/bin/python3.4m is present; as a result, anyone who was running version 1 of the image on the devel channel, and then upgraded, has the wrong (old) /usr/bin/python3.4 binary on their system.

This requires further investigation to understand both how the corruption of the delta happened, and how we can recover users on this channel who have applied the corrupted delta.

Status changed to 'Confirmed' because the bug affects multiple users.

I encounter a problem with a similar stack trace while trying to install a python 3 module named "secure-smtpd"

$ ./secure-smtpd-3.0.0/setup.py
Traceback (most recent call last):
  File "setup.py", line 3, in <module>
    from setuptools import setup, find_packages
  File "<virtualenv_path>/env_python3/lib/python3.4/site-packages/setuptools/__init__.py", line 11, in <module>
    from setuptools.extension import Extension
  File "<virtualenv_path>/env_python3/lib/python3.4/site-packages/setuptools/extension.py", line 8, in <module>
    from .dist import _get_unpatched
  File "<virtualenv_path>/env_python3/lib/python3.4/site-packages/setuptools/dist.py", line 18, in <module>
    from setuptools import windows_support
  File "<virtualenv_path>/env_python3/lib/python3.4/site-packages/setuptools/windows_support.py", line 2, in <module>
    import ctypes
  File "/usr/lib/python3.4/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: <virtualenv_path>/env_python3/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so: undefined symbol: _PyTraceback_Add

$ objdump -T /usr/bin/python3.4 | grep _PyTraceback_Add
0000000000485136 g DF .text 00000000000000f4 Base _PyTraceback_Add

Let me know if you need additional information.

$ objdump -T /usr/bin/python2 | grep _PyTraceback_Add

produces no result

hi guys
this error just happened to me when trying to import basestring from setuptools.compat (that also uses ctype).
just note that it just happened today after upgrading ubuntu 14.10 to 15.04
(this code was working before)

i got the following exception:

File "/home/danfromisrael/Dev/Projects/TodoApp/Server/Domain/Core/Jsonable.py", line 4, in <module>
    from setuptools.compat import basestring
  File "/home/danfromisrael/TodoAppEnv/lib/python3.4/site-packages/setuptools/__init__.py", line 11, in <module>
    from setuptools.extension import Extension
  File "/home/danfromisrael/TodoAppEnv/lib/python3.4/site-packages/setuptools/extension.py", line 8, in <module>
    from .dist import _get_unpatched
  File "/home/danfromisrael/TodoAppEnv/lib/python3.4/site-packages/setuptools/dist.py", line 18, in <module>
    from setuptools import windows_support
  File "/home/danfromisrael/TodoAppEnv/lib/python3.4/site-packages/setuptools/windows_support.py", line 2, in <module>
    import ctypes
  File "/usr/lib/python3.4/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so: undefined symbol: _PyTraceback_Add

hope it helps
Dan

Dan, the issue you describe is unrelated. Please file a separate bug for problems seen elsewhere than on the phone.

This is a critical infrastructure bug that should be considered a blocker for OTA-4, since the transition from OTA-3 to OTA-4 will include a delta that trips this bug.

This requires a server side

More information on comment #9:

<slangasek> because of bad handling of hardlinks in the delta generator
<slangasek> a minimal test case would be to generate a delta between two trees
            where a file whose link count is 2 has been updated [15:23]
<slangasek> and verify that both of the link targets are included in the delta
            (in a way that the upgrader can understand) [15:24]

Okay, so we have two bugs here related to hardlinks. Here's the scenario:

In the source tarball:

'a' is a regular file with some contents
'b' is a hardlink to 'a'

In the target tarball:
'a' contains different contents, but 'b' still hardlinks to 'a'

Bug #1: If, in the target tarball, 'a' appears before 'b', we end up with a change set of [del-b, mod-a]. That's clearly not right since the target will lose the 'b' file.

Bug #2: If, in the target tarball 'b' appears before 'a', we end up with a change set of [mod-a]. That's not correct because 'a' will get a new inode when the target is unpacked, but 'b' will still be pointing at the old inode, i.e. at the wrong file.

In both cases, we want to modify 'a' and write a new 'b' that points to the new 'a'. I'm thinking maybe we want a new (internal) diff generation action of 'rep' == 'replace' for 'b', so we'd see a change set in both cases of [mod-a, rep-b]

Hmm, reading generate_diff_tarball(), maybe we just need [mod-a, mod-b]

FYI the problematic delta is still available on system-image.ubuntu.com. The relevant files are:

old image (20141201): http://system-image.ubuntu.com/pool/ubuntu-fa74c536de07b74b0c7484c4e20ebc471a5965bc784b1a38face0b8e7598316a.tar.xz
new image (20150413): http://system-image.ubuntu.com/pool/ubuntu-631ce387d6c729c3da0ae9177d0c9d9a07f3e0dfb15b21b20771079322915d20.tar.xz
delta: http://system-image.ubuntu.com/pool/ubuntu-631ce387d6c729c3da0ae9177d0c9d9a07f3e0dfb15b21b20771079322915d20.delta-ubuntu-fa74c536de07b74b0c7484c4e20ebc471a5965bc784b1a38face0b8e7598316a.tar.xz

Old tarball:

$ tar --wildcards -tvf /srv/system-image.ubuntu.com/www/full/pool/ubuntu-fa74c536de07b74b0c7484c4e20ebc471a5965bc784b1a38face0b8e7598316a.tar.xz system/usr/bin/python3.4'*'
-rwxr-xr-x root/root 3120604 2014-11-14 19:14 system/usr/bin/python3.4m
hrwxr-xr-x root/root 0 2014-11-14 19:14 system/usr/bin/python3.4 link to system/usr/bin/python3.4m
$

New tarball:

$ tar --wildcards -tvf /srv/system-image.ubuntu.com/www/full/pool/ubuntu-631ce387d6c729c3da0ae9177d0c9d9a07f3e0dfb15b21b20771079322915d20.tar.xz system/usr/bin/python3.4'*'
-rwxr-xr-x root/root 2887488 2015-03-27 02:51 system/usr/bin/python3.4m
hrwxr-xr-x root/root 0 2015-03-27 02:51 system/usr/bin/python3.4 link to system/usr/bin/python3.4m
$

And in the unpacked delta:

$ grep usr/bin/python3.4 removed
system/usr/bin/python3.4m
$

And the contents of the delta:

$ tar --wildcards -tvf /srv/system-image.ubuntu.com/www/full/pool/ubuntu-631ce387d6c729c3da0ae9177d0c9d9a07f3e0dfb15b21b20771079322915d20.delta-ubuntu-fa74c536de07b74b0c7484c4e20ebc471a5965bc784b1a38face0b8e7598316a.tar.xz system/usr/bin/python3.4'*'
-rwxr-xr-x root/root 2887488 2015-03-27 02:51 system/usr/bin/python3.4m
$

So system/usr/bin/python3.4m is removed and then re-added - leaving system/usr/bin/python3.4 unmodified (and thus stale).

The fix has landed on the server, and I have confirmed on a test channel that it gives correct results when generating a delta between the current stable image and the current rc-proposed image:

$ ./bin/copy-image ubuntu-touch/stable/ubuntu ubuntu-touch/lp-1444347-test krillin 22
$ ./bin/copy-image ubuntu-touch/rc-proposed/ubuntu ubuntu-touch/lp-1444347-test krillin 180
$ tar tvf www/full/pool/ubuntu-1da961aa25eb6982f71fd433895d98d19b564b02f6f1ff20b4c830778f394016.delta-ubuntu-9c7a7516955ff8b2cc7d5a0053d45954e2d8fe0c4cfa97d4c3dca2bc0582405e.tar.xz | grep usr/bin/python3.4
-rwxr-xr-x root/root 2887488 2015-03-27 02:51 system/usr/bin/python3.4m
hrwxr-xr-x root/root 0 2015-03-27 02:51 system/usr/bin/python3.4 link to system/usr/bin/python3.4m
$

Marking this issue resolved.

For users who encountered this issue on the vivid devel channel, if you haven't already (and you probably have), please re-flash with the current image to get the correct image contents.

I've the same critical error but is thrown by Wand library. Currently I'm using django and a update has broken my python3.4. The result of command :
objdump -T /usr/bin/python3.4 | grep _PyTraceback_Add
is:
0000000000457e27 g DF .text 00000000000000cf Base _PyTraceback_Add

My currrent version of Ubuntu is :
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty

I did an apt-get upgrade on a Raspberry Pi running

Linux raspberry 3.18.0-25-rpi2 #26-Ubuntu SMP PREEMPT Sun Jul 5 06:46:34 UTC 2015 armv7l armv7l armv7l GNU/Linux

Now when I try to use IPython as a shell_plus in Django ./manage.py shell_plus --ipython I get the following error:
…
File "/home/ubuntu/.virtualenvs/Affinity-Django-Robot/lib/python3.4/site-packages/IPython/lib/inputhook.py", line 202, in decorator
    self.guihooks[toolkitname] = inst
AttributeError: 'InputHookManager' object has no attribute 'guihooks'

/manage.py shell_plus --plain functions as expected

I tried upgrading IPython but I could not because ctypes were missing. (I am assuming Python 3.4 is broken).

Django seems to be working as expected otherwise.

objdump -T /usr/bin/python3.4 | grep _PyTraceback_Add

0002912c g DF .text 0000008a Base _PyTraceback_Add

Also of note, the python used inside of the virtualenv is Python 3.4.0 (default, Jun 19 2015, 14:24:27)

The /usr/bin/python3 is Python 3.4.3 (default, Oct 14 2015, 21:23:51)

Inside of a Docker container the same Django app functions as expected when I run

./manage.py shell_plus --ipython

Linux 83b0b44f3023 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Python 3.4.3

output of objdump -T /usr/bin/python3.4 | grep _PyTraceback_Add

000000000042c7cf g DF .text 00000000000000cf Base _PyTraceback_Add