User data is not parsed correctly by Eucalyptus in some cases

Confirming this:
$ printf "%s\n%s\n" '#!/bin/sh' 'echo hello world" > /tmp/user-data.sh
$ euca-run-instances -k mykey --user-data-file /tmp/user-data.sh emi-220011A6 -t m1.small
$ euca-describe-instances
RESERVATION r-342805FE admin default
INSTANCE i-3BB70743 emi-220011A6 192.168.1.192 172.19.1.2 running mykey 0 m1.small 2009-10-26T15:21:07.472Z sm-zone-1a eki-3EE316B1 eri-225F1633
$ ssh -i mykey.priv ubuntu@192.168.1.192 'wget -q http://169.254.169.254/latest/user-data -O -'; echo
IyEvYmluL3NoCmVjaG8gaGVsbG8gd29ybGQK

The above should show plaintext.
$ dpkg-query --show "euca*"
euca2ools 1.0+bzr20091007-0ubuntu1
eucalyptus-cc 1.6~bzr931-0ubuntu7
eucalyptus-cloud 1.6~bzr931-0ubuntu7
eucalyptus-common 1.6~bzr931-0ubuntu7
eucalyptus-gl 1.6~bzr931-0ubuntu7
eucalyptus-java-common 1.6~bzr931-0ubuntu7
eucalyptus-javadeps
eucalyptus-sc 1.6~bzr931-0ubuntu7
eucalyptus-walrus 1.6~bzr931-0ubuntu7

Please also think of a workaround in ec2-init if fixing eucalyptus is no longer a possibility.

------------------------------------------------------------
revno: 942
committer: Neil
branch nick: 1.6
timestamp: Mon 2009-10-26 09:36:30 -0700
message:
  fixes #461156
------------------------------------------------------------

In my reproduction test I retrieve a string that is base64'd two times:

$ ssh -i mykey.priv ubuntu@192.168.0.230 'wget -q http://169.254.169.254/latest/user-data -O -'; echo
SXlFdlltbHVMM05vQ21WamFHOGdQVDA5UFQwOVBUMDlJRWhGVEV4UElEMDlQVDA50

Which base64-decodes to:
IyEvYmluL3NoCmVjaG8gPT09PT09PT09IEhFTExPID09PT09

Which base-64 decodes to:
#!/bin/sh
echo ========= HELLO =====

I believe I've verified that using euca-run-instances base64 encodes data 2 times rather than once.
For example, with euca2ools configured to run against ec2, I do:

$ euca-run-instances --user-data "hello world" --key ec2-keypair ami-ef00e386
$ ssh ${ec2_host} 'wget -q http://169.254.169.254/latest/user-data -O -'; echo
aGVsbG8gd29ybGQ=
$ ssh $ec2_host 'wget -q http://169.254.169.254/latest/user-data -O -' |
   base64 -d; echo
hello world

This is true for both '--user-data' and '--user-data-file'.

A bit more data...
I'm fairly sure that the correct thing to do here is just to remove the following two lines from main() in euca-run-instances:
| if user_data:
| user_data = base64.urlsafe_b64encode(user_data)

Also 'import base64' on line 37 could then be removed as this is the only place base64 is used.

The reason we dont need this encoding is that boto is doing it for us.

In euca-run-instances, the following are the lines that matter:

| if user_data:
| user_data = base64.urlsafe_b64encode(user_data)
| euca_conn = euca.make_connection()
| try:
| reservation = euca_conn.run_instances(image_id = image_id,
| min_count = min_count,
| max_count = max_count,
| key_name = keyname,
| security_groups = group_names,
| user_data = user_data,
...

euca.make_connection in euca2ools/euca2ools/__init__.py returns a connection that is created with:
   return boto.connect_ec2()

Then, we call euca_conn.run_instances. run_instances in boto/ec2/connection.py does:
| if user_data:
| params['UserData'] = base64.b64encode(user_data)

So, the user data is getting encoded twice.

Turns out it is an issue with euca2ools and not eucalyptus. reverting.

------------------------------------------------------------
revno: 943
committer: Neil
branch nick: 1.6
timestamp: Mon 2009-10-26 10:52:44 -0700
message:
  reverting invalid fix in revno 942 (was not a bug).
------------------------------------------------------------

See #461301

From duplicate bug 461301:
------------------------------------------------------------
revno: 222
committer: Neil
branch nick: euca2ools-1.0
timestamp: Mon 2009-10-26 11:00:47 -0700
message:
  fixes #461301
------------------------------------------------------------

From bug 461922 (which discusses a fix that did not make it into Ubuntu yet):
=====================================
After applying the [proposed] fix, euca-run-instances will fail depending on the user-data input. I found this while trying to provide a shell script as user data input. The bug is triggered by the --user-data parameter as well as providing user-data from a file using --user-data-file.

lindenm@dyn-220:~/cloud$ euca-run-instances -k markuskey -t c1.medium --user-data " << FO >" --kernel eki-245816A6 --ramdisk eri-05D01618 emi-092D1197
RESERVATION r-46BB06FB admin admin-default
INSTANCE i-47980954 emi-092D1197 0.0.0.0 0.0.0.0 pending markuskey 2009-10-27T12:37:51.99Z eki-245816A6 eri-05D01618

lindenm@dyn-220:~/cloud$ euca-run-instances -k markuskey -t c1.medium --user-data "<< FOO >" --kernel eki-245816A6 --ramdisk eri-05D01618 emi-092D1197
RESERVATION r-28AE050D admin admin-default
INSTANCE i-4EFD095C emi-092D1197 0.0.0.0 0.0.0.0 pending markuskey 2009-10-27T12:44:14.422Z eki-245816A6 eri-05D01618

lindenm@dyn-220:~/cloud$ euca-run-instances -k markuskey -t c1.medium --user-data " << FOO >" --kernel eki-245816A6 --ramdisk eri-05D01618 emi-092D1197
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden

I understand that euca-run-instances should work with any sort of user-data regardless of it's content. ec2-run-instances works flawlessly with eucalyptus in this case.

Note the used input strings:

"<< FOO >" (no space at the beginning, works)
" << FO >" (space, two character word, works)
" << FOO >" (space at the beginning, three character word, fails)
=================================

I am unable to reproduce this problem.

euca-run-instances -k mykey --kernel eki-0212156F --ramdisk eri-604B16B8 emi-97BE1397 -z mycluster2 --user-data " << FOO >" --addressing private
RESERVATION r-405607B1 admin admin-default
INSTANCE i-45F30812 emi-97BE1397 0.0.0.0 0.0.0.0 pending mykey 2009-10-27T17:06:34.508Z eki-0212156F eri-604B16B8

euca2ools revno 222, boto-1.8d.

Hmm. I was using the latest karmic euca2ools package (bzr200) w/ just euca-run-instances "fixed". If anything else related to user data handling changed in between, this might be the cause. I'll be able to look into that tomorrow.

I'm also unable to reproduce Markus's problems with --user-data ' << FOO >' and a patched euca2ools.

I've built a ppa build of euca2ools with the patch at https://launchpad.net/~smoser/+archive/ppa .

I tested that the patch works against ec2 with user data of " << FOO >", and that it fails against eucalyptus as Markus said.

The issue is that boto (used by euca2ools) is encoding the supplied user-data in connection.py:run_instances with:
| base64.urlsafe_b64encode(user_data)

For many input strings it is true that:
  base64.b64decode(base64.urlsafe_b64encode(str)) == str

For '<< FOO >' it is not. It will throw an error.

My guess is that eucalyptus is taking the request from euca2ools and doing a b64decode rather than urlsafe_b64decode on it. This is throwing error and being returned to the user as a parse error.

Thus, I'm fairly sure that eucalyptus will have to be changed to expect user_data to be encoded via urlsafe_b64decode() rather than b64decode().

urlsafe_b64encode is explained at http://docs.python.org/library/base64.html as:
| Encode string s using a URL-safe alphabet, which substitutes - instead of +
| and _ instead of / in the standard Base64 alphabet. The result can still contain =.

I suspect that the eucalyptus

I just verified that ec2-api-tools work with such user data when pointed at a eucalyptus server:
$ ec2-run-instances -k mykey --user-data " << FOO >" emi-247011C0
RESERVATION r-45F60808 admin admin-default
INSTANCE i-523209EF emi-247011C0 0.0.0.0 0.0.0.0 pending mykey 0
                m1.small 2009-10-27T19:37:32+0000 sm-zone-1a eki-41E516CA eri-253A1644 monitoring-false
$ euca-run-instances -k mykey --user-data " << FOO >" emi-247011C0
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden

$ dpkg-query --show euca2ools
euca2ools 1.0+bzr20091007-0ubuntu1.1~ppa

I also verified that inside the instance created by ec2-run-instances the meta-data server has the correct value for user-data.

I'm attaching a snippit from /var/log/eucalyptus/cloud-error.log, that occurs when you run the failing euca-run-instances command above.

Scott, sorry I don't understand what you are saying.

You said "I'm also unable to reproduce Markus's problems with --user-data ' << FOO >' and a patched euca2ools."

But you are still running into the "User authentication failed" issue?

I cannot reproduce this against the upsteam (current euca2ools or eucalyptus).

thanks
neil

On Tue, 27 Oct 2009, Neil Soman wrote:

> You said "I'm also unable to reproduce Markus's problems with --user-
> data ' << FOO >' and a patched euca2ools."

That should have said "*able* to reproduce" . I do see it.

To reproduce, you have to
a.) fix euca2ools so that it doesn't doubly encode user-data
    hand-patch the euca-run-instances, or use my ppa build

b.) run: ec2-run-instances -k mykey --user-data " << FOO >" ${EMI}

The result is:
| Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
| EC2ResponseError: 403 Forbidden
| Failure: 403 Forbidden

And the attached log message in cloud-error.log.

Yep, I confirm that it is a bug against eucalyptus (not euca2ools). I was using an older version of euca2ools that did double encoding of the user data.

thanks.

To fix this issue an SRU is needed for karmic:

1. fix euca-run-instances to only encode user-data once to mimic what ec2-run-instances does
2. fix eucalyptus soap|rest interface to not fail when user-data is sent by euca-run-instances.

Documented at <https://wiki.ubuntu.com/KarmicKoala/ReleaseNotes#UEC%20user-data%20not%20usable%20by%20guest%20instances>:

When user data is passed to an instance started with {{{euca-run-instances}}} (using either the {{{-d}}}, {{{--user-data}}} option or the {{{-f}}}, {{{--user-data-file}}} option), the data returned at {{{http://169.254.169.254/latest/user-data}}} will be base64-encoded. {{{ec2-init}}} is unable make use of this user data because it must be decoded before use. A fix for this issue is expected to be provided in a post-release update immediately after the Ubuntu 9.10 release. (Bug:461156)

Test case A:
A1 euca-run-instances -k $MYKEY --user-data "FOO" $EMI -t c1.medium
A2 ssh -i $MYKEYPRIV ubuntu@$IP 'wget -q http://169.254.169.254/latest/user-data -O -'; echo
Expected results: A2 should return "FOO"

Test case B:
B1 euca-run-instances -k $MYKEY --user-data " << FOO >" $EMI -t c1.medium
B2 ssh -i $MYKEYPRIV ubuntu@$IP 'wget -q http://169.254.169.254/latest/user-data -O -'; echo
Expected results: B2 should return " << FOO >"

With unpatched euca2ools and unpatched eucalyptus:
Test A fails with A2 returning "Rk9P" instead of "FOO"
Test B fails with B2 returning "IDw8IEZPTyA-" instead of " << FOO >"

With patched euca2ools (from smoser PPA) and unpatched eucalyptus:
Test A : PASS
Test B fails with B1 returning "403 Forbidden"

Note that python-boto does encode UserData using base64.b64encode (not urlsafe_b64encode as suggested in comment 13). The 403 Forbidden issue probably comes from URL decoding hitting special characters in HmacV2Handler.java, since they are no longer "protected" by double base64 encoding. Usage of canonicalString.replaceAll("\\+","%20") in there seems particularly relevant, since " << FOO >" translates to "IDw8IEZPTyA+"

> Note that python-boto does encode UserData using base64.b64encode
> (not urlsafe_b64encode as suggested in comment 13).

Thierry is correct. The commenter in comment 13 above must have been staring at a computer screen too long. boto uses base64.b64encode everywhere that i can see: http://code.google.com/p/boto/source/browse/trunk/boto/ec2/connection.py .

The "eucalyptus" part of this bug might well be bug 477776.

Thierry/Scott-

Are there proposed fixes for this issue, at this point?

@Thierry
- #21 the change of "+" is only for the purposes of computing the hmac, not processing the message contents.
- #23 the bug referenced is not related to this issue.

There look to be (at least) two issues here:
1) double base64'd data
2) authentication failure w/ certain user data strings

#1 was fixed in the euca2ools, afaict. #2 no longer manifests with any of the examples listed in this thread.

Is there still a pending issue here and, if so, is there a test case?

cheers.
chris

------------------------------------------------------------
revno: 1079
committer: decker <decker@personal-army>
branch nick: 1.6
timestamp: Fri 2009-11-20 23:33:29 -0800
message:
  fixes authentication problem w/ userdata lp:#461156
------------------------------------------------------------

Fix committed to lp:~ttx/eucalyptus/karmic-sru2
Test with: eucalyptus 1.6~bzr931-0ubuntu7.4~ppa1 in https://launchpad.net/~ttx/+archive/ppa

With patched euca2ools (from smoser PPA) and patched eucalyptus (from ttx PPA):
Test A : PASS
Test B : PASS

Eucalyptus part of the fix now committed to lp:~ubuntu-core-dev/eucalyptus/ubuntu-karmic

This bug was fixed in the package eucalyptus - 1.6.1~bzr1083-0ubuntu1

---------------
eucalyptus (1.6.1~bzr1083-0ubuntu1) lucid; urgency=low

  [ Dustin Kirkland ]
  * Merge upstream bzr revision 1082; the following bugs have been fixed
    upstream since the last merge:
    - LP: #378969 - private bug
    - LP: #404842 - init script fix
    - LP: #434283 - existing keys should be overwritten unconditionally
    - LP: #445990 - run instance will fail if no kernel or ramdisk specified
    - LP: #447457 - euca_conf --register-sc ... check the number of parameters
    - LP: #449874 - fix incorrect help text (--delete-nodes doesn't exist)
    - LP: #451795 - show registered images in elastic fox
    - LP: #454405 - return correct networkIndex values on describeInstances
    - LP: #456877 - init script fix
    - LP: #456878 - fix for libvirt xen driver
    - LP: #460085 - fix rampart memory leak
    - LP: #461156 - fix authentication problem w/ userdata
    - LP: #461394 - fix multiple concurrent snapshots on the same volume
    - LP: #461444 - fix memory leaks in NC getConsoleOutput and startup_thread
    - LP: #469984 - fix iptables rules issue
    - LP: #477776 - fix query string authentication
    - LP: #480783 - allow api connection over https
    - LP: #482249 - fix "Describe Regions"
    - LP: #484217 - create keypair should return an error if key exists
    - LP: #490623 - parse RFC 1123 formatted datetime
  * debian/control:
    - make all package lists one-per-line (makes changes henceforth more
      readable), sort lists
    - depend on rampart >= 1.3.0-0ubuntu6, which fixes some shared library
      installation issues
  * debian/patches/04-axis2c-1.6.0-rampart-1.3.0.patch: drop this patch,
    since Eucalyptus 1.6.1 natively supports axis2c 1.6.0 now
  * debian/eucalyptus-cloud.install,
    debian/eucalyptus-common.eucalyptus.upstart,
    debian/eucalyptus-java-common.install, debian/eucalyptus-sc.install,
    debian/eucalyptus-walrus.install: update static version number strings
    from "1.6-devel" to "1.6.1"; (we should really find a better way to do
    this)
  * debian/patches/03-DESTDIR.patch: ported forward for merge
 -- Dustin Kirkland <email address hidden> Tue, 01 Dec 2009 21:09:28 -0600

This bug was fixed in the package euca2ools - 1.0+bzr20091007-0ubuntu2

---------------
euca2ools (1.0+bzr20091007-0ubuntu2) lucid; urgency=low

  [ Scott Moser ]
  * Do not base64 encode user-data in euca-run-instances, as boto takes
    care of that (LP: #461156)
 -- Thierry Carrez <email address hidden> Wed, 02 Dec 2009 10:08:14 +0100

Accepted eucalyptus into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Accepted euca2ools into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Thierry-

Can you verify this package?

We need other fixes associated with this update here onsite at a partner.

Verification:
With karmic / karmic-updates : FAIL
With karmic-proposed (eucalyptus + euca2ools): PASS

This bug was fixed in the package euca2ools - 1.0+bzr20091007-0ubuntu1.1

---------------
euca2ools (1.0+bzr20091007-0ubuntu1.1) karmic-proposed; urgency=low

  * Do not base64 encode user-data in euca-run-instances, as boto takes
    care of that (LP: #461156)
 -- Scott Moser <email address hidden> Mon, 26 Oct 2009 20:38:04 -0400

This bug was fixed in the package eucalyptus - 1.6~bzr931-0ubuntu7.4

---------------
eucalyptus (1.6~bzr931-0ubuntu7.4) karmic-proposed; urgency=low

  [ Thierry Carrez ]
  * cluster/handlers.c: Cherrypick upstream r946: initialize axis2c in the CC
    for each NC client connection, to avoid rampart memory leak (LP: #460085)
  * clc/modules/wsstack/src/main/java/com/eucalyptus/ws/handlers/HmacV2Handler.java:
    Cherrypick upstream r1079: Fix authentication issue when using a euca2ools
    that doesn't double base64encode userdata (LP: #461156)

  [ Dustin Kirkland ]
  * debian/eucalyptus-cc.upstart, debian/eucalyptus-common.eucalyptus.upstart:
    support CLEAN=1 on start/stop/restart of eucalyptus/euclayptus-cc; export
    the CLEAN env variable in eucalyptus.init, and handle it in both the
    pre-start and post-stop sections of eucalyptus-cc, (LP: #491254)
 -- Dustin Kirkland <email address hidden> Wed, 02 Dec 2009 17:58:18 -0600

Fairly sure this is fix-released in Eucalyptus, right?

Should be, yes.