metadata server cannot be contacted when deploying new instances on EUC

Bug #765275 reported by Eric Williams
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
eucalyptus (Ubuntu)
Confirmed
Medium
Unassigned

Bug Description

## Issue

New instances deployed on EUC cannot contact the metadata server, failing with Connection refused.

## Environment

Distributor ID: Ubuntu
Description: Ubuntu Natty (development branch)
Release: 11.04
Codename: natty

ii eucalyptus-admin-tools 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Admin tools
ii eucalyptus-cc 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Cluster controller
ii eucalyptus-cloud 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Cloud controller
ii eucalyptus-common 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Common files
ii eucalyptus-gl 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Logging service
ii eucalyptus-java-common 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Common Java package
ii eucalyptus-sc 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Storage controller
ii eucalyptus-walrus 2.0.1+bzr1256-0ubuntu4 Elastic Utility Computing Architecture - Walrus (S3)
ii libeucalyptus-commons-ext-java 0.5.0-0ubuntu2 Eucalyptus commons external Java library

## Resolution

- reinstalling usually fixes the issue, but this is suboptimal

## Diagnostic Steps

euca-get-console-output shows the following:

lxcmount stop/pre-start, process 47
init: udev-fallback-graphics main process (256) terminated with status 1
init: plymouth main process (49) killed by SEGV signal
init: plymouth-splash main process (259) terminated with status 2
cloud-init start-local running: Mon, 18 Apr 2011 22:05:50 +0000. up 7.14 seconds
no instance data found in start-local
init: cloud-init-local main process (255) terminated with status 1
cloud-init start running: Mon, 18 Apr 2011 22:05:50 +0000. up 7.70 seconds
2011-04-18 22:05:50,859 - DataSourceEc2.py[WARNING]: waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id

2011-04-18 22:05:50,861 - DataSourceEc2.py[WARNING]: 22:05:50 [ 1/30]: url error [[Errno 111] Connection refused]

2011-04-18 22:05:51,864 - DataSourceEc2.py[WARNING]: 22:05:51 [ 2/30]: url error [[Errno 111] Connection refused]

2011-04-18 22:05:52,868 - DataSourceEc2.py[WARNING]: 22:05:52 [ 3/30]: url error [[Errno 111] Connection refused]

...

2011-04-18 19:55:09,908 - DataSourceEc2.py[WARNING]: 19:55:09 [26/30]: url error [[Errno 111] Connection refused]

2011-04-18 19:55:15,917 - DataSourceEc2.py[WARNING]: 19:55:15 [27/30]: url error [[Errno 111] Connection refused]

found data source: DataSourceEc2
mountall: Event failed

Opening http://169.254.169.254/2009-04-04/meta-data/instance-id from the cluster controller gives a 404 error, and not a "111 / Connection refused".

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: eucalyptus-cloud 2.0.1+bzr1256-0ubuntu4
ProcVersionSignature: Ubuntu 2.6.38-8.42-server 2.6.38.2
Uname: Linux 2.6.38-8-server x86_64
.etc.eucalyptus.eucalyptus.cc.conf: CC_NAME="disaster-cluster1"
Architecture: amd64
Date: Mon Apr 18 18:24:52 2011
ProcEnviron:
 LANGUAGE=en_US:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: eucalyptus
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Eric Williams (eric-canonical) wrote :
Revision history for this message
Boris Devouge (bdevouge) wrote :

Reinstalling fixes the issue on Maverick, but not fully tested on 11.04, which could be a different issue.

Revision history for this message
Torsten Spindler (tspindler) wrote :

We tried the following to overcome the problem, nothing was successful:

1) Restart eucalyptus-cc
2) Reboot front-end and node-controlle
3) Purge and reinstall eucalyptus-cc

Revision history for this message
Eric Williams (eric-canonical) wrote :

Attaching a tcpdump of the following scenario:

- new instance is started
- it loops through the attempts to get the metadata via http
- the wget fails with connection refused

thanks,
eric

Chuck Short (zulcss)
Changed in eucalyptus (Ubuntu):
importance: Undecided → Medium
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.