Ubuntu
cloud-init package

[SRU] apt_sources broken in 0.6.3-0ubuntu1.3 [regression]

Bug #1100491 reported by Adam Gandelman
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Fix Released
Critical
Scott Moser

Bug Description

[Impact]

cloud-init can no longer process cloud-config that specify a 'packages' value. This affects any tools (Juju, MAAS) that rely on cloud-init to install various packages as part of its first-boot. This seems to trigger when specifying apt_sources in cloud-init.

[Test Case]

Create a userdata file:

cat >>/tmp/ud.txt
#cloud-config
apt_sources:
- {source: 'ppa:openstack-ubuntu-testing/folsom-trunk-testing'}
apt_update: true
apt_upgrade: false
packages: [python-novaclient]
<<END

Boot a precise daily Ubuntu Cloud image, specifying the ud.txt as user-data:

euca-run-instances -k adam -t m1.tiny -f /tmp/ud.txt ami-04d

Allow instance to boot. Since MAAS provisiongs nodes on-the-fly, first boot will have the affected cloud-init version and the attached traceback in /var/log/cloud-init-output.log.

For cloud instances, its most likely running an older version of cloud-init and the cloud-config was processed okay on first boot. To test the affected version there, ssh to the machine:

# revert changes to system from initial cloud-init run.
$ sudo dpkg -P python-novaclient ;\
  sudo rm -rf /etc/apt/sources.list.d/openstack-ubuntu-testing-folsom-trunk-testing-precise.list ;\
  sudo rm -rf /var/lib/cloud/*
# Install cloud-init from precise-updates
$ sudo apt-get install cloud-init=0.6.3-0ubuntu1.3

# re-run cloud-init
$ sudo cloud-init start-local ; sudo cloud-init start; sudo cloud-init-cfg all config ; sudo cloud-init-cfg all final

Observe traceback ending in "TypeError: add_sources() takes at most 2 arguments (3 given)", the expected package was not reinstalled.

Apply fix to /usr/share/pyshared/cloudinit/CloudConfig/cc_apt_update_upgrade.py and re-run cloud-init. Works as
expected, package is reinstalled.

[Regression Potential]

Minimal, simple one-line.

[Original Report]

Noticed juju deployed services were failing to come up using Juju+MAAS and the newly released cloud-init SRU for 12.04. The node provisioned thru MAAS okay, but after first boot did not have the required agents installed. On further investigation, it looks like the required juju packages were never installed. /var/log/cloud-init-output.log shows the following traceback:

The key fingerprint is:
b4:83:ef:99:6e:65:15:a3:a3:a2:83:0b:f1:3c:62:7f root@test-08
The key's randomart image is:
+--[ECDSA 256]---+
| |
| o |
| . . o |
| o .o . |
|. . S. o |
| + ....o |
|o.+. . ..o |
|.oo.oE ..o |
| .o.. o= |
+-----------------+
Generating locales...
  en_US.UTF-8... up-to-date
Generation complete.
2013-01-16 16:32:56,919 - __init__.py[WARNING]: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/__init__.py", line 117, in run_cc_modules
    cc.handle(name, run_args, freq=freq)
  File "/usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/__init__.py", line 78, in handle
    [name, self.cfg, self.cloud, cloudinit.log, args])
  File "/usr/lib/python2.7/dist-packages/cloudinit/__init__.py", line 326, in sem_and_run
    func(*args)
  File "/usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/cc_apt_update_upgrade.py", line 75, in handle
    errors = add_sources(cloud, cfg['apt_sources'], params)
TypeError: add_sources() takes at most 2 arguments (3 given)

2013-01-16 16:32:56,919 - __init__.py[ERROR]: config handling of apt-update-upgrade, None, [] failed

2013-01-16 16:32:56,998 - cloud-init-cfg[ERROR]: errors running cloud_config [config]: ['apt-update-upgrade']
errors running cloud_config [config]: ['apt-update-upgrade']

Attached are /var/log/cloud-init.log and the MAAS provided user-data from /var/lib/cloud/instance/user-data.txt

See original description
Revision history for this message
Adam Gandelman (gandelman-a) wrote :
Revision history for this message
Adam Gandelman (gandelman-a) wrote :
Revision history for this message
Scott Moser (smoser) wrote :

reproduces with:
#cloud-config
apt_update: True
apt_sources:
 - source: "ppa:smoser/ppa"

Revision history for this message
Scott Moser (smoser) wrote :

fixed with:
$ diff -u /usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/cc_apt_update_upgrade.py.dist /usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/cc_apt_update_upgrade.py
--- /usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/cc_apt_update_upgrade.py.dist 2013-01-16 22:11:06.472490044 +0000
+++ /usr/lib/python2.7/dist-packages/cloudinit/CloudConfig/cc_apt_update_upgrade.py 2013-01-16 22:11:08.208488640 +0000
@@ -72,7 +72,7 @@
         params = mirrors
         params['RELEASE'] = release
         params['MIRROR'] = mirror
- errors = add_sources(cloud, cfg['apt_sources'], params)
+ errors = add_sources(cfg['apt_sources'], params)
         for e in errors:
             log.warn("Source Error: %s\n" % ':'.join(e))

Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Removing the first parameter in the call to add_sources() seems to fix the issue:

=== modified file 'CloudConfig/cc_apt_update_upgrade.py'
--- CloudConfig/cc_apt_update_upgrade.py 2013-01-16 22:05:40 +0000
+++ CloudConfig/cc_apt_update_upgrade.py 2013-01-16 22:10:42 +0000
@@ -72,7 +72,7 @@
         params = mirrors
         params['RELEASE'] = release
         params['MIRROR'] = mirror
- errors = add_sources(cloud, cfg['apt_sources'], params)
+ errors = add_sources(cfg['apt_sources'], params)
         for e in errors:
             log.warn("Source Error: %s\n" % ':'.join(e))

http://paste.ubuntu.com/1539261/

Scott Moser (smoser)
Changed in cloud-init (Ubuntu):
assignee: nobody → Scott Moser (smoser)
importance: Undecided → Critical
status: New → In Progress
Adam Gandelman (gandelman-a)
description: updated
summary: - cloud-init 0.6.3-0ubuntu1.3 failing to process juju-generated userdata
+ [SRU] cloud-init 0.6.3-0ubuntu1.3 failing to process juju-generated
+ userdata
Scott Moser (smoser)
description: updated
Revision history for this message
Chris Halse Rogers (raof) wrote : Please test proposed package

Hello Adam, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed
Revision history for this message
Adam Gandelman (gandelman-a) wrote : Re: [SRU] cloud-init 0.6.3-0ubuntu1.3 failing to process juju-generated userdata

Was able to test the proposed package on an EC2 instance using the test case in the SRU text in bug description. Log attached.

tags: added: verification-done
removed: verification-needed
Scott Moser (smoser)
summary: - [SRU] cloud-init 0.6.3-0ubuntu1.3 failing to process juju-generated
- userdata
+ [SRU] apt_sources broken in 0.6.3-0ubuntu1.3
summary: - [SRU] apt_sources broken in 0.6.3-0ubuntu1.3
+ [SRU] apt_sources broken in 0.6.3-0ubuntu1.3 [regression]
tags: added: regression-update
Revision history for this message
Scott Moser (smoser) wrote :

I think this should be fix-committed as it is in proposed.

Changed in cloud-init (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Scott Moser (smoser) wrote :

I'm attaching a user-data file that I've built to demonstrate this issue, and that it is resolved.
Simply:
 * launch an instance with the provided user-data
 * ssh to instance. wait for /home/ubuntu/<version>/ directory to contain 'cloud-init-output.log', and then reboot

On ssh in, you'll see a directory named for the cloud-init version that was present on boot.
Once that directory has 'cloud-init.log' in it, then you can reboot.

The next boot should have the next version of cloud-init in it (ie, if instance had 0.6.3-0ubuntu1.1, then 0.6.3-0ubuntu1.3 should be installed. If 0.6.3-0ubuntu1.3 was there , then -proposed should have been installed).

Just looking at the associated logs will show if there as an error.
I've verified using a daily image, that had 0.6.3-0ubuntu1.3 in it. (ebs/ubuntu-precise-daily-amd64-server-20130117)

Revision history for this message
Colin Watson (cjwatson) wrote :

Waiving the waiting period in -proposed since this is a critical regression.

Revision history for this message
Colin Watson (cjwatson) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.4

---------------
cloud-init (0.6.3-0ubuntu1.4) precise-proposed; urgency=high

  * lp-1100491-fix-broken-add-sources.patch: fix use of
    'apt_sources' in cloud-config. (LP: #1100491)
 -- Scott Moser <email address hidden> Wed, 16 Jan 2013 17:19:50 -0500

Changed in cloud-init (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Scott Moser (smoser) wrote :

Just to make sure, I've verified this is fixed in the latest daily of precise builds. Using the user-data attached above, ssh'd into instance, and saw the following in 0.6.3-0ubuntu1.4/output.log.

cloud-init 0.6.3-0ubuntu1.4
== sources.list.d ===
total 4
-rw-r--r-- 1 root root 178 Jan 18 13:57 ubuntu-server-ec2-testing-dev-testing-precise.list
== grep Traceback /var/log/cloud-init.log ==
== apt-cache policy smhello ==
smello:
  Installed: (none)
  Candidate: 0.3~ppa1
  Version table:
     0.3~ppa1 0
        500 http://ppa.launchpad.net/ubuntu-server-ec2-testing-dev/testing/ubuntu/ precise/main amd64 Packages
== removing proposed list ==
== Done ==

I just did that test in:
us-east-1 ami-d375feba canonical ebs/ubuntu-precise-daily-amd64-server-20130117.1

So the only precise / 12.04 images affected by this bug are the daily builds of 20130117. (fixed in 20130117.1 and later)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.