cloud-init

base64 user-data broken in OVF datasource

Bug #1619394 reported by Julian Vassev
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
Medium
Unassigned
cloud-init (Ubuntu)
Fix Released
Medium
Unassigned
Xenial
Fix Released
Medium
Unassigned

Bug Description

==== Begin SRU Template ====
[Impact]
Users of the OVF datasource can not provide user-data that is base64 encoded.

[Test Case]
$ rel=xenial
$ burl="http://cloud-images.ubuntu.com/daily/server/$rel/current/"
$ disk="$rel-server-cloudimg-amd64-disk1.img"
$ wget "$burl/$disk" -O "$disk"

# prepare the ovftransport.iso
$ git clone http://git.launchpad.net/cloud-init
$ ( cd cloud-init/doc/sources/ovf && ./make-iso ovf-env.xml.tmpl user-data --output - ) > ovftransport.iso

# create a temp disk for patching and patch in -proposed
$ qemu-img create -f qcow2 -b "$disk" disk.img
$ sudo mount-image-callback --system-resolvconf disk.img -- chroot _MOUNTPOINT_ \
    sh -c 'l=/etc/apt/sources.list.d/proposed.list;
       echo deb http://archive.ubuntu.com/ubuntu/ xenial-proposed main > "$l" &&
       apt-get -q update && apt-get install -qy cloud-init'

$ qemu-system-x86_64 -enable-kvm \
   -device virtio-net-pci,netdev=net00 -netdev type=user,id=net00 \
   -hda disk.img -cdrom ovftransport.iso -m 512 -curses -snapshot

# you will see an 'ovfdemo' login prompt, log in as root:passw0rd
# verify it worked by cat /var/lib/cloud/instance/user-data.txt

[Regression Potential]
very low chance for regression, a guest with user-data would be broken previously.

==== End SRU Template ====

I am using cloudint 0.7.7~bzr1256-0ubuntu1~16.04.1 on Ubuntu 16.04.1 LTS.

When I pass yaml-formatted user-data over the OVF datasource over the ISO transport the yaml string fails to be parsed.

I tracked this to minidom's inability to handle newlines in an attribute. The xml bellow works for coreos but breaks under cloud-init:

<PropertySection>
         <Property oe:key="user-data" oe:value="#cloud-config
---
users:
  - name: "test"
    passwd: "$1$NVdKo9MI$PlHasfYsufCHbP1Hh9TMz/"
    groups:
      - "sudo"
"/>
</PropertySection>

If I use base64-encoded user-data I get this error:

  Sep 01 12:07:43 sof2-lab3-dhcp371 cloud-init[3248]: 2016-09-01 12:07:43,854 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'I2Nsb3VkLWNvbmZpZwotLS0K'...'
  Sep 01 12:07:43 sof2-lab3-dhcp371 cloud-init[3248]: [CLOUDINIT] __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'I2Nsb3VkLWNvbmZpZwotLS0K'...'

Is there a way to pass user-data as a single-line string that doesn't confuse minidom?

See original description

Related branches

~smoser/cloud-init:bug/1619394
cloud-init Commiters: Pending requested
Diff: 123 lines (+87/-4)
2 files modified
cloudinit/sources/DataSourceOVF.py (+4/-4)
tests/unittests/test_datasource/test_ovf.py (+83/-0)
Revision history for this message
Scott Moser (smoser) wrote :

fwiw, i'm pretty sure it is invalid ovf to have newlines in the value of property.
minidom might be doing the right thing.

the fix for this in cloud-init is to allow you to pass in base64 encoded string in that user-data.

that path was broken in python3.

Julian put a branch to fix that at https://code.launchpad.net/~jvassev/cloud-init/+git/cloud-init/+ref/fix-b64decode-ovf .

I've put a branch up at https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/305496 with a unit test, and i'm going to pull that in now.

Changed in cloud-init:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Scott Moser (smoser) wrote :

fixed in 0.7.8.

Changed in cloud-init:
status: Confirmed → Fix Released
Scott Moser (smoser)
Changed in cloud-init (Ubuntu):
status: New → Fix Released
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Xenial):
status: New → In Progress
importance: Undecided → Medium
Scott Moser (smoser)
summary: - OVF datasource broken
+ base64 user-data broken in OVF datasource
Scott Moser (smoser)
description: updated
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Julian, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7-31-g65ace7b-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Julian Vassev (jvassev) wrote :

Hi Chris,
Upgrading to 0.7.7-31-g65ace7b-0ubuntu1~16.04.1 from -proposed fixes my issue.

Scott Moser (smoser)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Martin Pitt (pitti) wrote :

Hello Julian, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-1-g3705bb5-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Scott Moser (smoser) wrote :

verified as shown in SRU template.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.9 KiB)

This bug was fixed in the package cloud-init - 0.7.8-1-g3705bb5-0ubuntu1~16.04.1

---------------
cloud-init (0.7.8-1-g3705bb5-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream release 0.7.8.
  * New upstream snapshot.
    - systemd: put cloud-init.target After multi-user.target (LP: #1623868)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * debian/control: add Breaks of older versions of walinuxagent (LP: #1623570)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/control: fix missing dependency on python3-serial,
    and make SmartOS datasource work.
  * debian/cloud-init.templates fix capitalisation in template so
    dpkg-reconfigure works to select OpenStack. (LP: #1575727)
  * d/README.source, d/control, d/new-upstream-snapshot, d/rules: sync
    with yakkety for changes due to move to git.
  * d/rules: change PYVER=python3 to PYVER=3 to adjust to upstream change.
  * debian/rules, debian/cloud-init.install: remove install file
    to ensure expected files are collected into cloud-init deb.
    (LP: #1615745)
  * debian/dirs: remove obsolete / unused file.
  * upstream move from bzr to git.
  * New upstream snapshot.
    - Allow link type of null in network_data.json [Jon Grimm] (LP: #1621968)
    - DataSourceOVF: fix user-data as base64 with python3 (LP: #1619394)
    - remove obsolete .bzrignore
    - systemd: Better support package and upgrade. (LP: #1576692, #1621336)
    - tests: cleanup tempdirs in apt_source tests
    - apt config conversion: treat empty string as not provided. (LP: #1621180)
    - Fix typo in default keys for phone_home [Roland Sommer] (LP: #1607810)
    - salt minion: update default pki directory for newer salt minion.
      (LP: #1609899)
    - bddeb: add --release flag to specify the release in changelog.
    - apt-config: allow both old and new format to be present.
      [Christian Ehrhardt] (LP: #1616831)
    - python2.6: fix dict comprehension usage in _lsb_release. [Joshua Harlow]
    - Add a module that can configure spacewalk. [Joshua Harlow]
    - add install option for openrc [Matthew Thode]
    - Generate a dummy bond name for OpenStack (LP: #1605749)
    - network: fix get_interface_mac for bond slave, read_sys_net for ENOTDIR
    - azure dhclient-hook cleanups
    - Minor cleanups to atomic_helper and add unit tests.
    - Fix Gentoo net config generation [Matthew Thode]
    - distros: fix get_primary_arch method use of os.uname [Andrew Jorgensen]
    - Apt: add new apt configuration format [Christian Ehrhardt]
    - Get Azure endpoint server from DHCP client [Brent Baude]
    - DigitalOcean: use the v1.json endpoint [Ben Howard]
    - MAAS: add vendor-data support (LP: #1612313)
    - Upgrade to a configobj package new enough to work [Joshua Harlow]
    - ConfigDrive: recognize 'tap' as a link type. (LP: #1610784)
    - NoCloud: fix bug providing network-interfaces via meta-data.
      (LP: 1577982)
    - Add distro tags on config modules that should have it [Joshua Harlow]
    - ChangeLog: update changelog for previous commit.
    - add ntp config module [Ryan Harper]
    - SmartOS: more improvement...

Read more...

Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
James Falcon (falcojr) wrote :

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2715

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.