Eucalyptus

euca-run-instances should fail if you try to run more instances than you can (in a single security group)

Bug #546293 reported by C de-Avillez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Eucalyptus
Triaged
Low
Daniel Nurmi
eucalyptus (Ubuntu)
Triaged
Wishlist
Unassigned

Bug Description

Using the tamarind-based test rig, and ISO images from cdimage (20100323.2).

 I tried to run the maximum number of instances on the setup (using lucid-amd64-topol, 66 sessions) from a idling cloud. All failed.

Looking at the configuration, I found VNET_ADDRSPERNET="32". I then tried to start 31 instances from a clean system again: all instances are shown in Pending, then go to Terminated (may take about 10 minutes to happen).

Waited for the cache to clear, and tried starting 29 instances -- success. Tried adding more, got a message stating "FinishedVerify: Not enough addresses left in the network subnet assigned to requested group: default". Interesting that I saw no such message when I tried to start from idle either 66 or 31 sessions.

Anyway. Closed them via via 'euca-terminate-instances $(euca-describe-instances | egrep ^INSTANCE| awk '{print $2}')'.

Started 30 sessions from idle: 29 started, one was left in Pending for a while (then Terminated), no message like above displayed. When I tried to start one more session, I got the "not enough addresses..." message.

Seems related to bug 462140

C de-Avillez (hggdh2)
summary: - Cannot start more than 28 instances
+ Cannot start more than 29 instances
Revision history for this message
Etienne Goyer (etienne-goyer-outlands) wrote : Re: Cannot start more than 29 instances

Hi Carlos,

I think you are having the problem discussed at:

    https://answers.launchpad.net/ubuntu/+source/eucalyptus/+question/87145

Revision history for this message
C de-Avillez (hggdh2) wrote : Re: No instances started if euca-run-instance is run with bug -n

Hi Etyenne,

Yes, the basic symptom is the same. The difference is that *no* instances succeed when I run with a -n > 30. This is rather unexpected.

BTW, thank you, adjusted the bug title.

summary: - Cannot start more than 29 instances
+ No instances started if euca-run-instance is run with bug -n
Revision history for this message
Etienne Goyer (etienne-goyer-outlands) wrote :

It is probably the same root cause, though. Does "euca-run-instances -n 30 ..." works if you increase the value of VNET_ADDRSPERNET and do a clean restart? If yes, then it's not really a bug, although I guess euca2ools should say something meaningful about why no instance are being started.

i am not sure what the correct behavior should be in this case. Should Eucalyptus actually run instances until it runs out of private address in the security group, or should it reject the entire request (like it seems to be doing here). My guess is that it should do the later, and return that it does not have resources to complete the request (or something along that line). The Eucalyptus people would know for sure what the intended behavior is supposed to be.

C de-Avillez (hggdh2)
summary: - No instances started if euca-run-instance is run with bug -n
+ No instances started if euca-run-instance is run with a large -n
Revision history for this message
C de-Avillez (hggdh2) wrote : Re: No instances started if euca-run-instance is run with a large -n

I am not sure either. But we do have some options:

1. if rejecting all requests, it should print out a reason, and end immediately;
2. if accepting up to (VNET_ADDRSPERNET - 3), then it should print out messages for the refusals (and not even try to start them) (but I guess there is a timing issue here). This is what happens when you are already at the limit, and you try another euca-run-instance on the same group.

But to accept all requests and keep them all Pending for many minutes before marking all Terminated is plainly wrong. Note that this happens from an idle system, so (on the default) we *do* have 29 instances to run.

Revision history for this message
Mathias Gug (mathiaz) wrote : Re: No instances started if euca-run-instance tries to run a number instances bigger than the limit (default 29) per security group

Dan mentioned this was fixed in an upstream branch. But won't make it in the 1.6.2 branch (lucid).

Changed in eucalyptus (Ubuntu):
importance: Undecided → Wishlist
status: New → Triaged
summary: - No instances started if euca-run-instance is run with a large -n
+ No instances started if euca-run-instance tries to run a number
+ instances bigger than the limit (default 29) per security group
Daniel Nurmi (nurmi)
Changed in eucalyptus:
assignee: nobody → Daniel Nurmi (nurmi)
Dustin Kirkland  (kirkland)
summary: - No instances started if euca-run-instance tries to run a number
- instances bigger than the limit (default 29) per security group
+ euca-run-instances should fail if you try to run more instances than you
+ can (in a single security group)
Daniel Nurmi (nurmi)
Changed in eucalyptus:
status: New → Triaged
importance: Undecided → Low
Revision history for this message
Andy Grimm (agrimm) wrote :

This issue is now being tracked upstream at http://eucalyptus.atlassian.net/browse/EUCA-2687

Please watch that issue for further updates.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.