Backports can't build-depend on other backports

The recommendation to use the aptitude resolver would make a lot more sense if the sbuild in launchpad-buildd had such a thing. So, I guess we have three possible courses of action here:

1) update the sbuild in launchpad-buildd (planned, but not currently a priority) and use different resolvers for different pockets (seems a bit sketchy)
2) Flip automatic installation on for backports on the buildds (may cause other issues, if we end up upgrading things we don't want)
3) Hack the current sbuild in lp-buildd to recognize the "build-dep not met but available in backports" situation and DTRT.

Status changed to 'Confirmed' because the bug affects multiple users.

Current versions of sbuild (I'm looking at 0.62.5-1ubuntu2) have changed how the apt resolver works. It now creates a dummy package whose dependencies match the build-depends of the package being built, puts that dummy package into an apt repository, adds the apt-repository to the sources.list, and installs the dummy package using apt.

Using a resolver with that behavior would fix this bug, so we may not need to use a different resolver on different pockets. I think this makes Adam's (1) the right long-term solution.

But given recent discussion, I don't think people generally have well-formed expectations of how the backports pocket *should* work (see the most recent Tech Board meeting, for example). I would prefer that we override NotAutomatic on builds in the backport pocket quickly than fix the resolver slowly, since that would be consistent with past behavior on backports builds.

I think that's fine for a temporary fix, but I think long term, this should behave more like experimental where the build dependency needs to be expressed to prevent using unintended backported packages.

Actually I think Evan's description is what's wanted.

If the version from backports is needed to satisfy the dependency, then it ought to be pulled from there. I believe that's what apt will do.

The "apt" resolver in the version of sbuild currently used on LP isn't that intelligent. sbuild parses out the build-depends, and runs "apt-get install pkg1 pkg2 pkg3" - you can see it in the failed build log in the bug's description. It's not intelligent enough to go to a non-default apt source.

Makes sense because the apt changes to support NotAutomatic are in Natty
and LP runs on Lucid.

No, no, I don't think it's inherently an issue with the version of apt - this code is all running within the build chroot, and uses the chroot's apt. When sbuild passes off the build-depends to apt, apt has no awareness of specific version requirements, so it pulls using the default source instead of looking to sources that are pinned lower.

Once apt has finished pulling from its default sources, sbuild goes back and says "Hey, you screwed up, I don't have a new enough version of whatever" and bails.

So it sounds like upgrading sbuild in launchpad-buildd would be enough ­— i.e. the apt resolver in newer sbuilds will DTRT for all pockets. Is that correct?

upgraded to which version?

we recently took a patch from slangasek to support muiltarch; we could
fairly easily deploy another

Ah, that I haven't analysed. What is the relationship between sbuild in launchpad-buildd and upstream sbuild? It seems like it is not just a snapshot (or at least it is quite an old snapshot; upstream looks pretty different now) so a wholesale update would be quite difficult.

The change needed is to make the resolver more smart, as described in comment #3. Some relevant bits upstream are

  http://anonscm.debian.org/gitweb/?p=buildd-tools/sbuild.git;a=blob;f=lib/Sbuild/ResolverBase.pm;h=4ed99f5f21d0fc1241932184e51d6ed2ed3cdd33;hb=HEAD#l503

  http://anonscm.debian.org/gitweb/?p=buildd-tools/sbuild.git;a=blob;f=lib/Sbuild/AptResolver.pm;h=6356394dc787149d0136bdb54e050c63c784e565;hb=HEAD

It's a hacked up version of DSA sbuild as used on Debian's buildds in 2004.

I ported everything to a largely-unmodified distro version in <https://code.launchpad.net/~wgrant/launchpad/use-system-sbuild> and <https://code.launchpad.net/~wgrant/ubuntu/lucid/sbuild/extended-result>. But it relies on LP DDEB support, which is close but not there yet.

Escalated by Bryce:

 This is a significant limitation on what can be provided
to users via backports. Solving this directly improves the Ubuntu
product for end users who need to install newer software on the stable
Ubuntu release.

I should probably take this bug, as I keep having discussions about it out-of-band and, I suspect, I'll ultimately be deeply involved with making it happen anyway.

I'm now convinced that the correct solution is for the buildds to /ignore/ NotAutomatic and always use whatever's available.

For backports, I agree entirely that ignoring NotAutomatic is just lovely, and likely the right thing. I understand that this is also being discussed as a solution to turning on NotAutomatic for proposed, however.

The above solution is perfectly reasonable if we also bring britney into play for proposed->updates migration in stable series as we plan to do for proposed->release in the development series. Without that, though, we end up building against bits that we might then be kicking out of proposed. Of course, this happens today, and doesn't seem to be much of a big deal, we manually sort it out when we mess it up, but it would still be nicer to have automated installability checks to go with this whole plan.

That's probably right for the devel release, but not for backports.

On Friday, July 27, 2012 07:53:49 AM Adam Conrad wrote:
> For backports, I agree entirely that ignoring NotAutomatic is just
> lovely, and likely the right thing. I understand that this is also
> being discussed as a solution to turning on NotAutomatic for proposed,
> however.

For backports, dependencies from backports should only be used for building if
the version in -release doesn't satisfy the minimum version requirements (like
Debian Experimental, AIUI).

The reason to keep experimental packages as isolated from each other as possible is that they might get ripped out without notice and leave other bits uninstallable. Do we really have that same policy with backports? I really see no issues with just declaring that, like any other Ubuntu pocket, backports is internally consistent, and let it depend on itself across the board.

(I'm partially biased here, in that implementing the other bit is significantly more work, and not on the near-term TODO, but I honestly don't see the argument for keeping backports compartmentalised, since we don't deliver them in a compartmentalised fashion, but as part of a widely-populated repository)

I think just ignoring NotAutomatic is definitely better than the current
situation, so if the other's a lot harder and ignoring could get done soon, I
definitely don't want to block progress.

My theory is that given that support for backports is different than
release/updates/security, I think the pocket should be design to minimize the
amount of interdependency to that users only get what they explicitly asked
for plus the least additional depends from backports that can be reasonably
managed.

That's consistent with why we made backports NotAutomatic to begin with. I
don't think users should enable backports and then eat the entire pocket.
They should get the new stuff (plus updates to that stuff) they explicitly ask
for and ~nothing else.

I'm not implementing it though, so I will repeat for emphasis that ignoring is
WAY better than the current situation and I don't want perfect to block
better.

Bryce actually de-escalated this bug in favour of bug 192076 on 2012-01-10

I'd really like to start backporting Jenkins to older releases as it moves so quickly upstream; however as Jenkins is actually more than a single package this problem blocks me right now as I need to build-dep on updated dependencies of the core package.

In order to backport juju-core we'll need go 1.1, which is not a good candidate for SRUing back to precise. If we could backport the newer go, and build from that, backporting juju-core to precise would be possible.

+1 to Martin Packman's comment 23. Being able to have juju core in 12.04 backports would immensely help the juju-core in LTS release story.

-Thanks,
Antonio

An other example of impossible backport: https://bugs.launchpad.net/ubuntu/+source/hplip/+bug/1248188

This is a show stopper, and needs a fix asap.

As can be seen with the builds at https://launchpad.net/ubuntu/+source/hplip/3.13.9-1~ubuntu12.04.1 this has been fixed for the precise chroots, and I'm uploading the rest now.

(For the record, I went with the "just ignore NotAutomatic" solution, in the form of pinning backports to 500, the same as any other pocket... We can revisit this another time if we really need/want more fine-grained control)