cloud-archive on precise not pinned when juju calls apt-get upgrade

TL;DR

juju-core, when deploying a new precise unit, calls apt-get upgrade with the cloud-archive repository in place and not pinned, meaning existing precise packages can be upgraded to newer versions in the cloud-archive. This is happening right now with twisted, where the 11.1 version in precise is being upgraded to 13.2 that comes from the cloud-archive.

This does not happen with the bootstrap node, where apt-get upgrade is called with the cloud-archive already pinned.

Details follow.

Since https://bugs.launchpad.net/cloud-archive/+bug/1240667/comments/9 juju-core started adding a pinning configuration for the cloud-archive repository on precise.

When bringing up a new unit, this is the order of events, regarding that repository, its pinning and apt-get calls:

1. cloud-archive sources list file created:
2. apt-get upgrade called
3. pinning file for the cloud archive created

Here are timings taken from a cs:precise/ubuntu deployment using juju-core 1.20.7: http://pastebin.ubuntu.com/8368459/

The problem is that the apt-get upgrade call is made when the pinning is not yet in place. Currently, this means that, for example, twisted is upgraded from 11.1 to 13.2 (!) in precise. That's quite a leap for LTS customers.

I think the apt-get upgrade call should be made after the pinning is in place, or else you risk upgrading important packages by mistake.

In the bootstrap node I end up with twisted 11.1 installed as I should:

root@juju-lsci-andreas-machine-0:~# dpkg -l|grep twisted
ii python-twisted-bin 11.1.0-1ubuntu2 Event-based framework for internet applications
ii python-twisted-core 11.1.0-1ubuntu2 Event-based framework for internet applications
ii python-twisted-names 11.1.0-1 DNS protocol implementation with client and server
ii python-twisted-web 11.1.0-1 HTTP protocol implementation together with clients and servers

And the cloud-archive is there, pinned. Notice the candidate for python-twisted-core:
root@juju-lsci-andreas-machine-0:~# apt-cache policy python-twisted-core
python-twisted-core:
  Installed: 11.1.0-1ubuntu2
  Candidate: 11.1.0-1ubuntu2
  Version table:
     13.2.0-1ubuntu1~ctools1 0
        400 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/cloud-tools/main amd64 Packages
 *** 11.1.0-1ubuntu2 0
        500 http://lsci-1.clouds.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status

From the bootstrap --debug output, the order seems to be as I suggested above for the new unit case: add repository, pin, apt-get update, apt-get upgrade.

Found one one wrinkle, though: it creates the pinning file twice for some reason.

andreas@nsn7:~$ juju bootstrap --debug -v
2014-09-18 00:10:36 INFO juju.cmd supercommand.go:37 running juju [1.20.7-trusty-amd64 gc]
...
2014-09-18 00:11:56 DEBUG juju.cloudinit.sshinit configure.go:51 Running script on ubuntu@159.8.9.12: dump_file() {
    code=$?
...
echo 'Adding apt repository: deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main' >&9
add-apt-repository -y 'deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main'
printf '%s\n' 'Explanation: Pin with lower priority, not to interfere with charms
Package: *
Pin: release n=precise-updates/cloud-tools
Pin-Priority: 400
' > '/etc/apt/preferences.d/50-cloud-tools'
echo 'Running apt-get update' >&9
apt-get --option Dpkg::Options::=--force-confold --assume-yes update
echo 'Running apt-get upgrade' >&9
...
install -D -m 644 /dev/null '/etc/apt/preferences.d/50-cloud-tools'
printf '%s\n' 'Explanation: Pin with lower priority, not to interfere with charms
Package: *
Pin: release n=precise-updates/cloud-tools
Pin-Priority: 400
' > '/etc/apt/preferences.d/50-cloud-tools'

And when it's time to install mongo from the cloud archive, bootstrap is very specific in its command line:
Start-Date: 2014-09-18 00:12:57
Commandline: apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install --target-release precise-updates/cloud-tools mongodb-server

It selects the cloud archive to overcome the pinning.