Require a way to copy [P]PPA packages into Ubuntu

This is dependent on the work being done in https://blueprints.edge.launchpad.net/soyuz/+spec/soyuz-delayed-copies which is nearly complete.

archive.syncSource API is already able to copy sources (binaries and custom uploads) from private PPA directly to the ubuntu main archives (PRIMARY and PARTNER).

This feature is *only* available to members of the ubuntu-drivers and ubuntu-security teams atm.

Here is an example:

{{{
    ubuntu = lp.distributions['ubuntu']
    primary, partner = ubuntu.archives
    security_ppa = lp.people['ubuntu-security'].archive
    primary.syncSource(
        source_name='hello', version='2.2-3ubuntu4',
        from_archive=security_ppa, include_binaries=True,
        to_series='jaunty', to_pocket='Security')
}}}

Version conflicts and other copy mistakes will result in failures and consequently no changes.

On success (nothing is returned), It results in an ACCEPTED item in the corresponding upload queue (ubuntu/jaunty/+queue) containing the source in question, its successfully built binaries and custom uploads available in the original archive (partially built sources will result in new builds in the destination archive).

The accepted 'delayed-copies' will be processed during the hourly publication cycle, bugs mentioned in the changelog will be closed and private files will become public at this point. The queue item will be marked as DONE.

No email notification will be generated (missing feature, sorry).

Next step is extend the permissions to execute copies to all ubuntu uploaders, respecting their permissions.

This is related to bug 244163 and can't be fixed yet for the same reasons.

Don't we need to force rebuilds in the primary archive always except for security unembargos? Doing otherwise will trivially cause library mismatches, FTBFS scenarios and so on.

Yes. This can be rolled up into the Package Copy Job stuff quite trivially now, since that is already in use for syncs from Debian. We just need the extra UI selection and view code logic.

r15480 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15480) is part of this bug's fix.

This OOPSes when processing the PCJ:

Traceback (most recent call last):
  Module lazr.jobrunner.jobrunner, line 194, in runJobHandleError
    self.runJob(job, fallback)
  Module lp.services.job.runner, line 270, in runJob
    super(BaseJobRunner, self).runJob(IRunnableJob(job), fallback)
  Module lazr.jobrunner.jobrunner, line 162, in runJob
    job.run()
  Module lp.soyuz.model.packagecopyjob, line 495, in run
    self.attemptCopy()
  Module lp.soyuz.model.packagecopyjob, line 562, in attemptCopy
    unembargo=self.unembargo)
  Module lp.soyuz.scripts.packagecopier, line 686, in do_copy
    previous_version=old_version)
  Module lp.soyuz.adapters.notification, line 193, in notify
    bprs=bprs)
  Module lp.soyuz.adapters.notification, line 477, in get_upload_notification_recipients
    info = fetch_information(spr, bprs, changes)
  Module lp.soyuz.adapters.notification, line 631, in fetch_information
    changesfile = spr.aggregate_changelog(previous_version)
  Module lp.soyuz.model.sourcepackagerelease, line 649, in aggregate_changelog
    for block in Changelog(changelog.read()):
  Module lp.services.librarian.model, line 171, in read
    self.open(timeout=timeout)
  Module lp.services.librarian.model, line 161, in open
    self._datafile = self.client.getFileByAlias(self.id, timeout)
  Module lp.services.librarian.client, line 476, in getFileByAlias
    return self._connect_read(url, try_until, aliasID)
  Module lp.services.librarian.client, line 491, in _connect_read
    raise LookupError(aliasID)
LookupError: 103960041

I'll need to figure out what's going on here; I assume it's either a missing transaction.commit or passing the wrong (private) objects to notify.

Archive.syncSource (the delayed-copy path) still works. So we can *probably* deploy this anyway, but the OOPS needs to be fixed ASAP.

Also, non-unembargoing Archive.copyPackage calls are unaffected.

And unembargo-package.py is unaffected, since it doesn't pass send_email=True; it's only PCJs that do that. So, in summary, the only broken thing is the new API which must be explicitly selected, making this safe to deploy despite the problems.

r15486 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15486) is part of this bug's fix.

r15486 seems to have cleared this up. The only mild oddity is that the BinaryPackageBuild is still on a URL for the private archive (https://dogfood.launchpad.net/~cjwatson/+archive/dogfood-private/+build/3459155), but it's actually public; this appears to be normal. Publication went off without a hitch as far as I can tell.

Yes, those URLS are normal:
https://launchpad.net/ubuntu/+source/nova/2011.3-0ubuntu6.8
https://launchpad.net/~ubuntu-security/+archive/ppa/+build/3568729

Apart from removing the old copy-package.py script, I think the only thing that's missing here is a way to copy packages into the Ubuntu primary archive using the UI. Of course, as mentioned, this requires care since we should only allow including binaries in certain circumstances. This isn't urgent in any case since the API is now sufficient for us.

r15772 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15772) is part of this bug's fix.