support adding region/availability to mirror selection

Bug #1037727 reported by Scott Moser
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Fix Released
High
Unassigned
Precise
Fix Released
High
Unassigned

Bug Description

=== Begin SRU Information ===
[Impact]
 * Currently, local mirror selection is done specifically for EC2 instances
   in the DataSource code. cloud-init will select
  'us-east-1a.ec2.archive.ubuntu.com' as a mirror in EC2. However, in other
  clouds, there is no way to get a local mirror selected.

  The change here is to allow mirrors to be specified in configuration based
  on templates that can include the "availability-zone". This makes the EC2
  code generic and allows a cloud provider to have cloud-init select a local
  mirror simply by presenting 'availability-zone' in the metadata.

  With this change, if a cloud provider puts 'region-1.mycloud' into their
  datasource metadata as the availability-zone, cloud-init will select
  a mirror at 'region-1.mycloud.clouds.archive.ubuntu.com'. By default,
  Canonical's dns will return that as a wildcard for archive.ubuntu.com,
  but the information is availble now in the dns request to specify
  the address of a local mirror.

[Test Case]
 * This is really a feature backport. There is no specific bug here.
   However, the behavior change is easily seen by running an instance on a
   openstack cloud. Previously, the mirror selected would be
   'archive.ubuntu.com'. With the change in place, the mirror will be:
   'nova.clouds.archive.ubuntu.com' as by default openstack provides the
   string 'nova' as the availability-zone.

[Regression Potential]
 * Mirror selection will change. It is possible that users relying on
   the old behavior will have to modify something.

[Other Information]
 * We want this behavior brought back to 12.04 so that LTS images can "just work"
   to select a local mirror simply by a change in DNS records. It will enable
   large clouds to provide local mirrors easily.

=== End SRU Information ===

currently, the ec2 datasource has some code that selects the mirror location by looking at the availability-zone.
I think this is generally a pretty good idea. I'd like to extend that to other datasources, and not limit to ".ec2.archive.ubuntu.com".

The goal would be to basically check for a mirror at:
  <availability-zone>[.<cloud>][dns-subdomain]

dns subdomain would default to '.cloud.archive.ubuntu.com'.

if the datasource provided the 'cloud' value, we'd use that, but otherwise just use availability-zone.

This would fit in with what we're currently doing on ec2, but would extend to others.
I'd just treat the "cc-direction-[0-9]" availability-zones specially, to know that this is an ec2 one. Then basically append ".ec2" in that case.

Related branches

Scott Moser (smoser)
Changed in cloud-init (Ubuntu):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Eric Hammond (esh) wrote :

I think you mean <region> and not <availability-zone> since availability zone names mean different things for different AWS accounts.

Revision history for this message
Scott Moser (smoser) wrote :

you're right. for amazon, 'availability-zone' there is 'region'.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.0~bzr633-0ubuntu1

---------------
cloud-init (0.7.0~bzr633-0ubuntu1) quantal; urgency=low

  * New upstream snapshot.
    * support creating users on boot. remove requirement for a 'ubuntu'
      user to be previously present in image. (LP: #1028503)
    * add experimental apt_reboot_if_required flag to reboot if necessary
      after upgrade or package install (LP: #1038108)
    * improve mirror selection for a distro:
      * support arm mirrors (LP: #1028501)
      * support seeding security mirror (LP: #1006963)
      * support dns mirrors including availability-zone reference
        (LP: #1037727)
    * include a "None" datasource so items like ssh host key generation
      occur if there is no other metadata service. (LP: #906669)
    * print authorized_keys for users to the console (LP: #1010582)
    * Add RHEVm and vSphere support as datasource AltCloud [Joseph VLcek]
 -- Scott Moser <email address hidden> Thu, 23 Aug 2012 01:06:34 -0400

Changed in cloud-init (Ubuntu):
status: Triaged → Fix Released
Scott Moser (smoser)
Changed in cloud-init (Ubuntu Precise):
status: New → Triaged
importance: Undecided → High
Scott Moser (smoser)
description: updated
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Scott, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Precise):
status: Triaged → Fix Committed
tags: added: verification-needed
Revision history for this message
Scott Moser (smoser) wrote :

I verified by getting an image (amd64 20121210), mounting it loopback, modifying /etc/resolv.conf then:

export DEBIAN_FRONTEND=noninteractive
echo "deb $mirror $rel-proposed main universe" > "/etc/apt/sources.list.d/$rel-proposed.list"
{
apt-get update --assume-yes --quiet &&
apt-get install cloud-init --assume-yes --quiet --option Dpkg::Options::=--force-confold
} </dev/null

Launched an instance of that.
Then:
$ ec2metadata --availability-zone
nova
$ grep "^deb .*ubuntu.*$(lsb_release -sc) main" /etc/apt/sources.list
deb http://nova.clouds.archive.ubuntu.com/ubuntu/ precise main
$ sed -n '/system_info/,$p' /etc/cloud/cloud.cfg
system_info:
   package_mirrors:
     - arches: [i386, amd64]
       failsafe:
         primary: http://archive.ubuntu.com/ubuntu
         security: http://security.ubuntu.com/ubuntu
       search:
         primary:
           - http://%(ec2_region)s.ec2.archive.ubuntu.com/ubuntu/
           - http://%(availability_zone)s.clouds.archive.ubuntu.com/ubuntu/
         security: []
     - arches: [armhf, armel, default]
       failsafe:
         primary: http://ports.ubuntu.com/ubuntu-ports
         security: http://ports.ubuntu.com/ubuntu-ports

tags: added: verification-done
removed: verification-needed
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Hello Scott, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-done
tags: added: verification-needed
Scott Moser (smoser)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.3

---------------
cloud-init (0.6.3-0ubuntu1.3) precise-proposed; urgency=low

  * debian/patches/lp-1070345-landscape-restart-after-change.patch,
    debian/patches/lp-1066115-landscape-install-fix-perms.patch:
    fix missing or incorrect imports (LP: #1070345, LP: #1066115).

cloud-init (0.6.3-0ubuntu1.2) precise-proposed; urgency=low

  * debian/patches/lp-978127-maas-oauth-fix-bad-clock.patch: fix usage of
    oauth in maas data source if local system has a bad clock (LP: #978127)
  * debian/cloud-init.preinst: fix bug where user data scripts re-ran on
    upgrade from 10.04 versions (LP: #1049146)
  * debian/patches/lp-974509-detect-dns-server-redirection.patch: detect dns
    server redirection and disable searching dns for a mirror named
    'ubuntu-mirror' (LP: #974509)
  * debian/patches/lp-1018554-shutdown-message-to-console.patch: write a
    message to the console on system shutdown. (LP: #1018554)
  * debian/patches/lp-1066115-landscape-install-fix-perms.patch: install
    landscape package if needed which will ensure proper permissions on config
    file (LP: #1066115).
  * debian/patches/lp-1070345-landscape-restart-after-change.patch: restart
    landscape after modifying config (LP: #1070345)
  * debian/patches/lp-1073077-zsh-workaround-for-locale_warn.patch: avoid
    warning when user's shell is zsh (LP: #1073077)
  * debian/patches/rework-mirror-selection.patch: improve mirror selection by:
    * allowing region/availability-zone to be part of mirror (LP: #1037727)
    * making mirror selection arch aware (LP: #1028501)
    * allow specification of a security mirror (LP: #1006963)
 -- Scott Moser <email address hidden> Thu, 13 Dec 2012 12:16:56 -0500

Changed in cloud-init (Ubuntu Precise):
status: Fix Committed → Fix Released
Revision history for this message
Colin Watson (cjwatson) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.