upstream version string updates require packaging branch changes or manual assignment

Even just being able to grab a tag would be amazing...

bzr tag 2.9.5-1ppa1
bzr push lp:mycode

recipe: {lasttag:mycode} or latesttag or ...

Then the user can pick what branch (probably packaging branch) that they tag. Of course that would imply that the tag is exactly what's needed for the version string. Example would be.. "2.3.4-9ppa1" or "2.3.9" or "2.3.9-1" would work - but "foo bar" would not work.

I think this is a patch that I could perhaps create.... I'll try but no guarantees what so ever...

On Sun, 2010-12-05 at 18:31 +0000, Michael Lustfield wrote:
> Even just being able to grab a tag would be amazing...
>
> bzr tag 2.9.5-1ppa1
> bzr push lp:mycode
>
> recipe: {lasttag:mycode} or latesttag or ...
>
> Then the user can pick what branch (probably packaging branch) that they
> tag. Of course that would imply that the tag is exactly what's needed
> for the version string. Example would be.. "2.3.4-9ppa1" or "2.3.9" or
> "2.3.9-1" would work - but "foo bar" would not work.
>
> I think this is a patch that I could perhaps create.... I'll try but no
> guarantees what so ever...
That's a very different issue though. You can already specify an
explicit revision to check out in a recipe, and what you are looking for
would (if I understand correctly) be covered by a new revspec
("lasttag:") in Bazaar.

Cheers,

Jelmer

> That's a very different issue though. You can already specify an
> explicit revision to check out in a recipe, and what you are looking for
> would (if I understand correctly) be covered by a new revspec
> ("lasttag:") in Bazaar.

You can use lasttag: in a recipe? What I had in my head was to avoid having to
update the recipe so you only need to update the relevant branches (including a
tag) and everything else magically works.

On Sun, 2010-12-05 at 19:28 +0000, Michael Lustfield wrote:
> > That's a very different issue though. You can already specify an
> > explicit revision to check out in a recipe, and what you are looking for
> > would (if I understand correctly) be covered by a new revspec
> > ("lasttag:") in Bazaar.
>
> You can use lasttag: in a recipe? What I had in my head was to avoid having to
> update the recipe so you only need to update the relevant branches (including a
> tag) and everything else magically works.
Ah, sorry. I thought you meant that you wanted to build the latest
tagged revision rather than tip ("Even just being able to grab a tag
would be amazing...")

I agree that having a {lasttag} variable in the recipe's version string
would be useful, but it still doesn't allow you to talk about e.g. alpha
revisions, where the final version hasn't been committed yet (so can't
be tagged).

It will also only work for a subset of upstreams. Samba's tags are named
e.g. release-4-0-0alpha13. {lasttag} would still not be very useful in
that case.

Cheers,

Jelmer

If repeatable "debianized tree output" is one of the design goals of the build recipe concept then surely what we are asking for here is impossible? I'd still like to explore solutions, though, but first I have a question:

Can somebody elaborate on why repeatable source packages are so important and how running the code in a launchpad hosted sandbox is not sufficient?

(see also https://bugs.launchpad.net/launchpad/+bug/608450/comments/6 )

I have been doing some research into this and I found out some
information that might be pertinent:

   * The versions numbers of binary packages are allowed to
     differ from the version numbers of their source package -
     http://wiki.debian.org/binNMU

   * Several people have tried, over many years, to use the
     substvars mechanism in "critical" fields such as the package
     name and the version number. There are some
     documented "wontfix" "wishlist" bugs against dpkg preventing
     this from working properly. I have not tried this myself to
     see if it has been fixed yet. -
     http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=66336 (and
     see also the merged bugs)

   * The substvars mechanism is invoked through debian/rules and
     thus theoretically the build dependencies are already
     installed.

My proposal would be to check (and fix if necessary) the use of
substvars to set the version number of DEBIAN/control. In order to
satisfy concerns about recreating the package, I think there are
a few options. My favourite is to have Launchpad refuse to
publish [dynamically named/numbered] packages unless the build job
URL (e.g. https://code.launchpad.net/~jwal/+archive/wkhtmltopdf/+buildjob/1970431
) is included in a new field called "Build-Job:" in the binary
package control file. The URL could be passed through from the
recipe to the debian/rules file as an environment variable.

Hi James,

Thanks for your comments.

On Sat, 2011-01-29 at 13:52 +0000, James Ascroft-Leigh wrote:
> I have been doing some research into this and I found out some
> information that might be pertinent:
>
> * The versions numbers of binary packages are allowed to
> differ from the version numbers of their source package -
> http://wiki.debian.org/binNMU
binNMU's are intended for no-change rebuilds of existing packages,
they're not a wild card for having divergent source and binary package
versions. Also, binNMU's are not supported in Ubuntu (we don't allow
binary uploads). Allowing non-upload rebuilds is on the wishlist though,
but not implemented (bug 245594).

> * Several people have tried, over many years, to use the
> substvars mechanism in "critical" fields such as the package
> name and the version number. There are some
> documented "wontfix" "wishlist" bugs against dpkg preventing
> this from working properly. I have not tried this myself to
> see if it has been fixed yet. -
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=66336 (and
> see also the merged bugs)

> * The substvars mechanism is invoked through debian/rules and
> thus theoretically the build dependencies are already
> installed.
I'm not sure I understand what you mean here? The build dependencies
are installed before substvars or debian/rules come into play at all
at the moment. Or is that what you're saying?

> My proposal would be to check (and fix if necessary) the use of
> substvars to set the version number of DEBIAN/control. In order to
> satisfy concerns about recreating the package, I think there are
> a few options. My favourite is to have Launchpad refuse to
> publish [dynamically named/numbered] packages unless the build job
> URL (e.g. https://code.launchpad.net/~jwal/+archive/wkhtmltopdf/+buildjob/1970431
> ) is included in a new field called "Build-Job:" in the binary
> package control file. The URL could be passed through from the
> recipe to the debian/rules file as an environment variable.
The concern with recreating packages is broader than just packages built
on Launchpad, it also covers local builds (which can't have a build job
URL).

Cheers,

Jelmer

Thanks for the quick response Jelmer. I was thinking that the build job link could be any URL. This would mean that a local build could have a build job URL like "file:///home/jwal/.bzr-recipe-logs/mypackage/1970-01-01T00:00:00Z.log" or "https://mycompany.example.com/buildbot/build/1234".

Regards,

James

On Sat, 2011-01-29 at 14:50 +0000, James Ascroft-Leigh wrote:
> Thanks for the quick response Jelmer. I was thinking that the build job
> link could be any URL. This would mean that a local build could have a
> build job URL like "file:///home/jwal/.bzr-recipe-
> logs/mypackage/1970-01-01T00:00:00Z.log" or
> "https://mycompany.example.com/buildbot/build/1234".
If the URL varies then that is one of the things that's not
deterministic. The same manifest should always result in the exact same
source package (every single byte, so the checksums match).

Cheers,

Jelmer

There is nothing preventing the build job URL from being included in the manifest as well. The point is that the build job URL is the first object allocated in the entire process. Any subsequent artefact including the manifest, log files, source package and binary packages can contain it.

It's the version number of the binary packages that cannot be included in the manifest because the manifest is generated too early in the process. That's fine, right? If you have a link back to the logs then using purely deterministic processes becomes less important.

Thanks,

James

On Sat, 2011-01-29 at 16:43 +0000, James Ascroft-Leigh wrote:
> There is nothing preventing the build job URL from being included in the
> manifest as well. The point is that the build job URL is the first
> object allocated in the entire process. Any subsequent artefact
> including the manifest, log files, source package and binary packages
> can contain it.

> It's the version number of the binary packages that cannot be included
> in the manifest because the manifest is generated too early in the
> process. That's fine, right? If you have a link back to the logs then
> using purely deterministic processes becomes less important.
We don't want to change the overall behaviour of the package building
process (that's for another day), so the package source versions and the
binary versions should stay the same.

I don't really understand how having a link back to the logs helps. Can
you explain? What would you expect bzr-builder to do to create a source
package from a recipe?

Cheers,

Jelmer

We currently rely on the version number of the package acting as an identifier to help find the source code for the binary package. If we are generating the version number during the build process (e.g. substvars) then we lose control of this. Putting the buildjob URL in the package instead restores confidence in the ability to get back to the source code. In fact, it makes me much more confident.

There might already be a better identifier that I am not aware of. If that is the case, apologies for the spam.

On Sat, 2011-01-29 at 19:03 +0000, James Ascroft-Leigh wrote:
> We currently rely on the version number of the package acting as an
> identifier to help find the source code for the binary package. If we
> are generating the version number during the build process (e.g.
> substvars) then we lose control of this. Putting the buildjob URL in
> the package instead restores confidence in the ability to get back to
> the source code. In fact, it makes me much more confident.
The problem is that the source package also has a version string, which
needs to be known before the build process kicks off at all, that
version string should match the version string of the binary package.

We also need to be able to generate the (self-contained) source package
bit for bit again from just the manifest afterwards.

Cheers,

Jelmer

You say that the version string in the source package must be known before the build process kicks off at all. You also say that the package source versions and the binary versions should stay the same. Those two statements combined seem to me to prevent us from fixing this bug. Is there no chance that we can relax these constraints?

Regards,

James

On Sat, 29 Jan 2011 19:26:30 -0000, Jelmer Vernooij <email address hidden> wrote:
> The problem is that the source package also has a version string, which
> needs to be known before the build process

Do you mean binary build process, or recipe build process here?

Thanks,

James

On Sat, 2011-01-29 at 21:37 +0000, James Westby wrote:
> On Sat, 29 Jan 2011 19:26:30 -0000, Jelmer Vernooij <email address hidden> wrote:
> > The problem is that the source package also has a version string, which
> > needs to be known before the build process
> Do you mean binary build process, or recipe build process here?
the source package build process (so halfway through the recipe build
process I guess?) - e.g. before being able to run 'debuild -S'.

Cheers,

Jelmer

On Sat, 2011-01-29 at 21:37 +0000, James Westby wrote:
> On Sat, 29 Jan 2011 19:26:30 -0000, Jelmer Vernooij <email address hidden> wrote:
> > The problem is that the source package also has a version string, which
> > needs to be known before the build process
> Do you mean binary build process, or recipe build process here?
the source package build process (so halfway through the recipe build
process I guess?) - e.g. before being able to run 'debuild -S'.

Cheers,

Jelmer

So, I don't think this was covered - to use it in LP, it can't run arbitrary code; thats why run is prohibited. Python code is still arbitrary.