euca-get-console-output returns one single line

I see this too. For the record, the error is not with the instance or instance kernel itself (as we have seen on ec2). The files in /var/lib/eucalyptus/instances/admin/${INSTANCE_ID}/console.log does exist and is populated, its contents just dont get back to the user.

If you have a setup where this is happening, could you see if there are any errors in nc.log related to this? Normally, it just logs the invocation of GetConsoleOutput, but perhaps there are errors there?

Yes, I got at least one on this last test. From the test script output:

2010-08-19 15:30:08,377 USER admin:DEBUG Running bash command: ['-c', 'euca-get-console-output --config=users/admin//eucarc i-62B50ACF']
(...)
2010-08-19 15:30:08,644 INSTANCE i-62B50ACF:WARNING Console output: START
2010-08-19 15:30:08,644 INSTANCE i-62B50ACF:WARNING i-62B50ACF
2010-08-19 15:30:08,644 INSTANCE i-62B50ACF:WARNING 2010-08-19T19:30:08.62Z
2010-08-19 15:30:08,644 INSTANCE i-62B50ACF:WARNING
2010-08-19 15:30:08,644 INSTANCE i-62B50ACF:WARNING
2010-08-19 15:30:08,645 INSTANCE i-62B50ACF:WARNING Console output: END
2010-08-19 15:30:08,645 INSTANCE i-62B50ACF:DEBUG Executing: euca-terminate-instances - ['i-62B50ACF']

And from the NC where it was running:

nc.log:[Thu Aug 19 15:30:08 2010][024891][EUCAINFO ] doGetConsoleOutput() invoked (id=i-62B50ACF)
nc.log:[Thu Aug 19 15:30:08 2010][024891][EUCAERROR ] cannot open '/var/lib/eucalyptus/instances//admin/i-62B50ACF/console.log' read-only
nc.log:[Thu Aug 19 15:30:08 2010][024891][EUCADEBUG ] doDescribeInstances(): instanceId=i-62B50ACF publicIp=0.0.0.0 privateIp=172.19.4.8 mac=D0:0D:62:B5:0A:CF vlan=10 networkIndex=8

Hum. That's weird.

Oh, I get it (from a currently-running instance):

ubuntu@soncoya:/var/lib/eucalyptus/instances/admin$ ls -l ./i-401D07A6
total 519652
-rw-rw---- 1 root root 20702 2010-08-19 17:32 console.log
-rw-r--r-- 1 root root 2179989504 2010-08-19 17:32 disk
-rw------- 1 eucalyptus eucalyptus 714696 2010-08-19 17:31 instance-checkpoint
-rw-r--r-- 1 root root 4475680 2010-08-18 15:32 kernel
-rw-r--r-- 1 eucalyptus eucalyptus 894 2010-08-19 17:31 libvirt.xml
ubuntu@soncoya:/var/lib/eucalyptus/instances/admin$

'console.log is root:root, 660 -- so eucalyptus, *unless* rootwrapped, cannot read it.

On Thu, 19 Aug 2010, C de-Avillez wrote:

> Oh, I get it (from a currently-running instance):
>
> ubuntu@soncoya:/var/lib/eucalyptus/instances/admin$ ls -l ./i-401D07A6
> total 519652
> -rw-rw---- 1 root root 20702 2010-08-19 17:32 console.log
> -rw-r--r-- 1 root root 2179989504 2010-08-19 17:32 disk
> -rw------- 1 eucalyptus eucalyptus 714696 2010-08-19 17:31 instance-checkpoint
> -rw-r--r-- 1 root root 4475680 2010-08-18 15:32 kernel
> -rw-r--r-- 1 eucalyptus eucalyptus 894 2010-08-19 17:31 libvirt.xml
> ubuntu@soncoya:/var/lib/eucalyptus/instances/admin$

Note, also, that disk and kernel are root owned there. On my system, in
the one instance I have running, 'kernel' was eucalyptus owned.

I'll watch the fix here, but the code I added for bug 611144 writes a file
in that directory named 'loader' from inside 'gen_kvm_libvirt_xml', which
is run as root, so that file also has root:root ownership.

Still, this suggests something changed. I do not mind the console log being root:root, but why 660 as permissions? Also, was console.log also owned by root before (I cannot test 1.6.2 right now)?

Nothing in Eucalyptus changed in this regard between 1.6.2 and 2.0.0, I believe. On two other KVM-based distros the console.log file, which is produced by KVM, is readable to Eucalyptus. We can add a patch to chown or chmod the file via euca_rootwrap every time prior to reading it, but that is a hack. Ideally, we would figure out why this is happening on Maverick. Could apparmor profile for KVM (or libvirt?) have something to do with this? (I agree that root ownership of "disk" and "kernel" is surprising and not consistent with other distros - those files are written by the NC directly.)

A discussion on IRC (#ubuntu-server) pointed to:

* taking out libvirt's ./debian/patches/9008-run-as-root-by-default.patch as a possible solution (even more due to the fact that this patch is now a delta with Debian)
* adding eucalyptus to the kvm group.

I am testing it now.

To summarize the IRC discussion:

libvirt uses a stacked security driver. The one at the bottom is always on and is the DAC driver. MAC drivers such as the AppArmor one used in Ubuntu sit on top of the DAC driver. The DAC driver looks at /etc/libvirt/qemu.conf for the user and group to run qemu/kvm as and as of libvirt 0.8.3 the DAC driver will consistently chown files to the user and group as defined in /etc/libvirt/qemu.conf. The DAC driver changes started to appear in earlier releases of the 0.8 series, and Debian started to use them during the Lucid cycle. Since Karmic, libvirt uses the AppArmor security driver and therefore qemu/kvm is confined more thoroughly by AppArmor than with the DAC driver and it was deemed too risky to run kvm/qemu as non-root since the libvirt code was not as well tested. So in Lucid, we changed qemu.conf back to running as root. Now because of 0.8.3's behavior of unconditionally chowning what qemu/kvm needs access to, disks and the console.log are chowned to 'root:root', which is why the euca user doesn't have access to the console.log.

The proper fix is to drop the 9008-run-as-root-by-default.patch patch and adding eucalyptus to the 'kvm' group. In addition to fixing this bug, it has the side benefits of reducing the Debian delta slightly and providing better protection for when the AppArmor driver is turned off.

I tested running as non-root with QRT and it is ok. I had some funky logic in there regarding save/restore tests that had to be adjusted so the libvirt-qemu:kvm user could write to the save directory I created. After fixing that, it works great with and without AppArmor enabled. I also went ahead and updated QRT to its 'libvirt-aa-secdriver.sh' tests with:
- root:root in qemu.conf and AppArmor enabled (the current Ubuntu default): PASS
- libvirt-qemu:kvm in qemu.conf and AppArmor enabled (the proposed default for maverick): PASS
- root:root in qemu.conf and AppArmor disabled: PASS
- libvirt-qemu:kvm in qemu.conf and AppArmor disabled (the current Debian default): PASS

Per discussion in IRC, server team voted to upload. Since my fingers are in the pie, assigning libvirt task to me.

This bug was fixed in the package libvirt - 0.8.3-1ubuntu7

---------------
libvirt (0.8.3-1ubuntu7) maverick; urgency=low

  * debian/patches/series: per Ubuntu Server team, run qemu/kvm as non-root
    and comment out 9008-run-as-root-by-default.patch. This has now received
    significant testing in Debian, adds a good security benefit for people
    who disable AppArmor and fixes the libvirt portion of LP: #619843. With
    this patch removed, libvirt will default to the Debian configure arguments
    and run qemu/kvm VMs as 'libvirt-qemu:kvm'.
  * debian/README.Debian: adjusted for the above
 -- Jamie Strandboge <email address hidden> Tue, 24 Aug 2010 12:58:51 -0500

OK. I also set it manually (right now). Looking at the instances directories, I see a mix of ownerships. Why?

ubuntu@mabolo:/var/lib/eucalyptus/instances/admin$ ls -lR
.:
total 28
drwxr-xr-x 2 eucalyptus eucalyptus 4096 2010-08-24 20:26 i-3878063D
drwxr-xr-x 2 eucalyptus eucalyptus 4096 2010-08-24 20:26 i-3DDB07C2
drwxr-xr-x 2 eucalyptus eucalyptus 4096 2010-08-24 20:26 i-3E240779
drwxr-xr-x 2 eucalyptus eucalyptus 4096 2010-08-24 20:26 i-4133084D
drwxr-xr-x 2 eucalyptus eucalyptus 4096 2010-08-24 20:27 i-44150778
drwxr-xr-x 2 eucalyptus eucalyptus 4096 2010-08-24 20:27 i-4628084F
drwxr-xr-x 2 eucalyptus eucalyptus 4096 2010-08-24 20:26 i-4D8108F1

./i-3878063D:
total 825080
-rw-rw---- 1 root root 22715 2010-08-24 20:27 console.log
-rw-r--r-- 1 root root 21507342336 2010-08-24 20:27 disk
-rw------- 1 eucalyptus eucalyptus 714696 2010-08-24 20:26 instance-checkpoint
-rw-r--r-- 1 root root 4401520 2010-08-24 17:40 kernel
-rw-r--r-- 1 eucalyptus eucalyptus 895 2010-08-24 20:26 libvirt.xml

./i-3DDB07C2:
total 539728
-rw-rw---- 1 libvirt-qemu kvm 19908 2010-08-24 20:27 console.log
-rw-r--r-- 1 libvirt-qemu kvm 2179989504 2010-08-24 20:27 disk
-rw------- 1 eucalyptus eucalyptus 714696 2010-08-24 20:26 instance-checkpoint
-rw-r--r-- 1 libvirt-qemu kvm 4401520 2010-08-24 17:40 kernel
-rw-r--r-- 1 eucalyptus eucalyptus 894 2010-08-24 20:26 libvirt.xml

./i-3E240779:
total 539780
-rw-rw---- 1 libvirt-qemu kvm 19753 2010-08-24 20:27 console.log
-rw-r--r-- 1 libvirt-qemu kvm 2179989504 2010-08-24 20:27 disk
-rw------- 1 eucalyptus eucalyptus 714696 2010-08-24 20:26 instance-checkpoint
-rw-r--r-- 1 libvirt-qemu kvm 4401520 2010-08-24 17:40 kernel
-rw-r--r-- 1 eucalyptus eucalyptus 894 2010-08-24 20:26 libvirt.xml

./i-4133084D:
total 4300
-rw-r--r-- 1 eucalyptus eucalyptus 4401520 2010-08-24 17:40 kernel

./i-44150778:
total 659948
-rw-rw---- 1 root root 22255 2010-08-24 20:26 console.log
-rw-r--r-- 1 root root 10769924096 2010-08-24 20:26 disk
-rw------- 1 eucalyptus eucalyptus 714696 2010-08-24 20:26 instance-checkpoint
-rw-r--r-- 1 eucalyptus eucalyptus 894 2010-08-24 20:26 libvirt.xml

./i-4628084F:
total 533264
-rw-rw---- 1 eucalyptus eucalyptus 0 2010-08-24 20:27 console.log
-rw-r--r-- 1 eucalyptus eucalyptus 2179989504 2010-08-24 20:27 disk
-rw-r--r-- 1 eucalyptus eucalyptus 4401520 2010-08-24 17:40 kernel

./i-4D8108F1:
total 583796
-rw-rw---- 1 libvirt-qemu kvm 17124 2010-08-24 20:27 console.log
-rw-r--r-- 1 libvirt-qemu kvm 5401214976 2010-08-24 20:26 disk
-rw------- 1 eucalyptus eucalyptus 714696 2010-08-24 20:26 instance-checkpoint
-rw-r--r-- 1 libvirt-qemu kvm 4401520 2010-08-24 17:40 kernel
-rw-r--r-- 1 eucalyptus eucalyptus 894 2010-08-24 20:26 libvirt.xml
ubuntu@mabolo:/var/lib/eucalyptus/instances/admin$

Carlos,

libvirt will only change the permissions for files it needs to, and only on demand. Therefore, it looks like i-4D8108F1, i-3E240779, and i-3DDB07C2 were all started after you changed qemu.conf and restarted libvirt. A reboot of the guest is not enough because a new qemu/kvm process is not launched. It looks like i-3878063D and i-44150778 (ie the ones with root:root ownership of the disk and console.log) were either rebooted or launched before libvirt was fully reloaded. i-4628084F is a mystery to me-- it certainly isn't anything libvirt did (libvirt doesn't know about the 'eucalyptus' user unless you configured qemu.conf to use it).

On Fri, 27 Aug 2010, Dmitrii Zagorodnov wrote:

> ** Changed in: eucalyptus
> Status: New => Invalid

Dmitrii,
   This is, in fact "invalid" in eucalyptus, but be careful, as the
solution you have in place will not work once distributions pick up later
versions of libvirt. So, for "eucalyptus" (as opposed to "ubuntu"),
you're going to either have to document how to fix it, or come up with
another solution, because you're going to see this more and more.

Roger that, Scott. Thanks!

Dave,

You changed this to fix-committed ... What change did you make? Did you add eucalyptus to the kvm group, or what?

@Dustin, Yes

For future reference, this can be seen by either viewing the changelog of the development branch (~ubuntu-virt) or noting the "Linked branch" feature in the bug report.

HTH

eucalyptus (2.0+bzr1239-0ubuntu1) maverick; urgency=low

  [Dave Walker (Daviey)]
  * New upstream stable release, -r1239 Fixes:
    - euca_conf --register-cluster is not idempotent. (LP: #628328)
    - unexpected errors after a sequence of tests (LP: #622818)
  * debian/control:
    - Recommend bittorrent|bittornado for -walrus package, enabling S3
      bittorent API support. (LP: #613535)
    - Raise the minimum version of libvirt-bin needed to eucalyptus-nc.
  * debian/eucalyptus-nc.postinst: Add eucalyptus user to the kvm group to
    enable user to utilise console logs. (LP: #619843)
  * debian/patches/18-priv_security.patch: Added vgscan to util/wrappers.conf,
    as it requires escalated privileges. (LP: #628291)
  * debian/patches/06-UEC-webinterface.patch (and 3 images): Updated and
    refreshed to better reflect Ubuntu branding. (LP: #631451)
  * debian/patches/25-uec-logo-png.patch: Due to a bug in dpkg, we need to use
    a different filename for the logo.png.
  * debian/eucalyptus-{java-,}common.{preinst,postinst}: Update the internal
    database and keys on upgrade from 10.04.

  [Scott Moser]
  * fix determination of 'is_multiboot'.
 -- Dave Walker (Daviey) <email address hidden> Fri, 10 Sep 2010 12:18:02 +0100