Launchpad itself

Bug #527245
Comment #3

Comment 3 for bug 527245

Revision history for this message

Colin Watson (cjwatson) wrote on 2015-07-03:

I'd really like to fix this (speaking as a frequent Haskell stack uploader ...), but it's harder than it seems.

Just drilling down through the uninstallable stack can produce rather inaccurate build-deps, because a non-trivial case of unsatisfiability can be resolved in several different ways (for example, A Build-Depends: B Depends: C with C missing might be resolved by C appearing or by B changing so that it no longer depends on C). What Debian does is to analyse the whole package set on the equivalent of every retry-depwait run. For just Ubuntu I think that would be tractable, but, unlike Debian, Launchpad has to consider PPAs too. A dose-builddebcheck run on something the size of Ubuntu takes about eight seconds here for one series/architecture; a rough-estimate query suggests that retry-depwait needs to consider something on the order of 3000 archives, and presumably multiple series and architectures within many of those.

So, we can't quite do exactly what Debian does. Some thoughts that come to mind:

* Since all PPAs today are based on the Ubuntu archive, most of the work is shared. In theory, dose-builddebcheck could be refactored to consider many archives and recalculate only what's changed. (I haven't profiled it, but perhaps most of the time is spent parsing the large baseline Packages files and not in resolving each set of build-dependencies on top of that?)
* We could have two modes for retry-depwait, one of which is used for primary archives and one for PPAs. This is pretty inelegant and we normally like to avoid that kind of thing, but it might be a reasonable stopgap.

That also leaves the question of when this work is done. If I understand wanna-build correctly, Debian puts brand-new builds into the Needs-Build state first, and any retry ("give-back") goes to BD-Uninstallable first. The state for new builds is OK, but Launchpad users are used to retries being almost immediate and I think they'd be annoyed by retries taking a minute or so to go through dose-builddebcheck. So I think it would be more sensible to retain the behaviour where we dispatch both new builds and retries to buildds as soon as we can, but for any builds that end up dep-waiting have retry-depwait fill in a new column on BinaryPackageBuild with the processed dose-builddebcheck output for that build. Future retry-depwait runs would retry builds if they vanish from dose-builddebcheck's list of failures.

Once this is in place, we could change the buildd code to declare dep-wait any time apt-get's resolver fails; maybe continue to provide an approximate analysis where we can but otherwise you'd have to wait for the next retry-depwait run before you got a full analysis.

With the exception of the dose-builddebcheck refactoring which my OCaml really isn't up to, the rest of this seems tractable. I think it would depend on whether we're willing to include the special case for primary archives to make the performance workable.

I'd really like to fix this (speaking as a frequent Haskell stack uploader ...), but it's harder than it seems.

Just drilling down through the uninstallable stack can produce rather inaccurate build-deps, because a non-trivial case of unsatisfiability can be resolved in several different ways (for example, A Build-Depends: B Depends: C with C missing might be resolved by C appearing or by B changing so that it no longer depends on C).  What Debian does is to analyse the whole package set on the equivalent of every retry-depwait run.  For just Ubuntu I think that would be tractable, but, unlike Debian, Launchpad has to consider PPAs too.  A dose-builddebcheck run on something the size of Ubuntu takes about eight seconds here for one series/architecture; a rough-estimate query suggests that retry-depwait needs to consider something on the order of 3000 archives, and presumably multiple series and architectures within many of those.

So, we can't quite do exactly what Debian does.  Some thoughts that come to mind:

* Since all PPAs today are based on the Ubuntu archive, most of the work is shared.  In theory, dose-builddebcheck could be refactored to consider many archives and recalculate only what's changed.  (I haven't profiled it, but perhaps most of the time is spent parsing the large baseline Packages files and not in resolving each set of build-dependencies on top of that?)
 * We could have two modes for retry-depwait, one of which is used for primary archives and one for PPAs.  This is pretty inelegant and we normally like to avoid that kind of thing, but it might be a reasonable stopgap.

That also leaves the question of when this work is done.  If I understand wanna-build correctly, Debian puts brand-new builds into the Needs-Build state first, and any retry ("give-back") goes to BD-Uninstallable first.  The state for new builds is OK, but Launchpad users are used to retries being almost immediate and I think they'd be annoyed by retries taking a minute or so to go through dose-builddebcheck.  So I think it would be more sensible to retain the behaviour where we dispatch both new builds and retries to buildds as soon as we can, but for any builds that end up dep-waiting have retry-depwait fill in a new column on BinaryPackageBuild with the processed dose-builddebcheck output for that build.  Future retry-depwait runs would retry builds if they vanish from dose-builddebcheck's list of failures.

With the exception of the dose-builddebcheck refactoring which my OCaml really isn't up to, the rest of this seems tractable.  I think it would depend on whether we're willing to include the special case for primary archives to make the performance workable.