Too easy for people to not use merge-upstream

Traceback (most recent call last):
  File "./import_package.py", line 1030, in <module>
    sys.exit(main(args[0]))
  File "./import_package.py", line 987, in main
    possible_transports=possible_transports)
  File "./import_package.py", line 928, in handle_collisions
    use_time_from_changelog=True)
  File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py", line 1555, in import_package
    timestamp=timestamp, author=author)
  File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py", line 1430, in _do_import_package
    version)
  File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py", line 1378, in upstream_parents
    pull_branch.revid_of_upstream_version(pull_version)
  File "/srv/package-import.canonical.com/new/scripts/plugins/builddeb/import_dsc.py", line 703, in revid_of_upstream_version
    return self.upstream_branch.tags.lookup_tag(tag_name)
  File "/usr/lib/python2.5/site-packages/bzrlib/tag.py", line 102, in lookup_tag
    raise errors.NoSuchTag(tag_name)
bzrlib.errors.NoSuchTag: No such tag: upstream-1.2.3

This has now failed for some reason.

Thanks,

James

Also affects apache2 import: "No such tag: upstream-2.2.15"

I agree that #3 is the best option. Any thoughts on how to do that (i.e. make it harder not to do the good path)?

This same traceback seems to sometimes happen when using merge-package to merge a new release from Debian. Take a look at lp:ubuntu/bzr-gtk It looks to me like I did every thing right (though it was a long time ago now). I merge in revision 13.2.8 revid:<email address hidden> which was a new upstream release
committed by Bazaar Package Importer <email address hidden> to the Debian branch, but the importer has been failing with "bzrlib.errors.NoSuchTag: No such tag: upstream-0.98.0" That's the version the importer itself pulled in...

If I go back in and manually push that tag will it make the branch usable? I just did a merge of bzr-gtk, and found it ironic that I had to use grab-merge.

libnotify4 branches require repair:
Upstream version 0.7.1 was imported erroneously specifying 1.7.1 as the version. Therefore, please:

bzr tag -d lp:ubuntu/natty/libnotify4 -r tag:upstream-1.7.1 upstream-0.7.1
bzr tag -d lp:ubuntu/oneiric/libnotify4 -r tag:upstream-1.7.1 upstream-0.7.1
bzr tag -d lp:ubuntu/natty/libnotify4 --delete upstream-1.7.1
bzr tag -d lp:ubuntu/oneiric/libnotify4 --delete upstream-1.7.1

and requeue the package.

bzr-gtk branches require repair:
As Andrew surmises, adding the missing tag across all relevant branches will fix this import. Please:

for d in lucid maverick natty oneiric; do bzr tag -d lp:ubuntu/$d/bzr-gtk -r <email address hidden> upstream-0.98.0; done

and requeue the package.

policykit-gnome branches require repair:
Please add the missing upstream tag:

bzr tag -r revid:<email address hidden> upstream-0.9.2 -d lp:ubuntu/lucid/policykit-gnome
bzr tag -r revid:<email address hidden> upstream-0.9.2 -d lp:ubuntu/maverick/policykit-gnome

and requeue the package.

fetchmail branches require repair:
Please add the missing upstream tag:

for d in maverick natty oneiric; do bzr tag -d lp:ubuntu/$d/fetchmail -r revid:<email address hidden> upstream-6.3.16; done

and requeue the package.

Thanks Max. I've run those commands and requeued those packages (bzr-gtk, policykit-gnome, fetchmail).

Oops, forgot libnotify4. Done that now too.

Thanks. Next up is gjs:

This one is slightly interesting, as an Ubuntu developer has attempted to manually commit the current oneiric package into the oneiric branch.

Unfortunately, this has been done without any upstream revision at all, which makes the importer very upset. I don't see any way around this short of manually uncommitting this manual revision:

bzr uncommit bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/oneiric/gjs

and removing the tag pointing to the uncommitted revision:

bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/oneiric/gjs \
  --delete 0.7.14-1ubuntu1

and then fixing the missing upstream tags which were the underlying issue:

for d in lucid maverick natty oneiric; do
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/gjs \
  -r <email address hidden> \
  upstream-0.5
done

and then requeueing the package.

For the record, here the diff between the "old" (manual, incorrect import of 0.7.14-1ubuntu1) and the "new" (result of running the importer locally):

Only in new: .pc
diff -ru old/gi/function.c new/gi/function.c
--- old/gi/function.c 2011-05-18 13:20:10.000000000 +0100
+++ new/gi/function.c 2011-05-18 13:20:20.000000000 +0100
@@ -646,6 +646,23 @@
             gboolean arg_failed;

             g_assert_cmpuint(next_rval, <, function->js_out_argc);
+#if G_BYTE_ORDER == G_BIG_ENDIAN
+ switch (return_tag) {
+ case GI_TYPE_TAG_INT8:
+ return_value.v_int8 = return_value.v_int;
+ break;
+ case GI_TYPE_TAG_UINT8:
+ return_value.v_uint8 = return_value.v_uint;
+ break;
+ case GI_TYPE_TAG_INT16:
+ return_value.v_int16 = return_value.v_int;
+ break;
+ case GI_TYPE_TAG_UINT16:
+ return_value.v_uint16 = return_value.v_uint;
+ default:
+ break;
+ }
+#endif
             arg_failed = !gjs_value_from_g_argument(context, &return_values[next_rval],
                                                     &return_info, (GArgument*)&return_value);
             if (arg_failed)
diff -ru old/gjs-internals-1.0.pc.in new/gjs-internals-1.0.pc.in
--- old/gjs-internals-1.0.pc.in 2011-05-18 13:20:10.000000000 +0100
+++ new/gjs-internals-1.0.pc.in 2011-05-18 13:20:20.000000000 +0100
@@ -11,7 +11,6 @@
 mozjslibdir=@FIREFOX_JS_LIBDIR@

 Cflags: -I${includedir}/gjs-1.0 @JS_EXTRA_CFLAGS@
-Libs: -Wl,--rpath=${mozjslibdir}
 Requires: gjs-1.0 gobject-introspection-1.0 @JS_PACKAGE@

 Name: gjs-internals-1.0

Hello,

Is it possible to have a look to these bzr branches which required repair I think: lp:ubuntu/cairo-dock, lp:ubuntu/natty/cairo-dock, lp:ubuntu/cairo-dock-plug-ins, lp:ubuntu/natty/cairo-dock-plug-ins ?
I guess Cairo-Dock bzr branches for Maverick need repair too but it's less important (there is a new bugs fixed version for Natty and Oneiric)

aurora branches require fixup:

Here the problem is that manual commits have been pushed to the branch, which claim to be upstream version updates, but in fact only update the debian/changelog.

Thus, we need to uncommit the bogus commits and remove their tags, then requeue the package:

for d in maverick natty oneiric; do
bzr uncommit -r tag:1.6.0-0ubuntu3 bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/aurora
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/aurora --delete 1.6.3-0ubuntu1
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/aurora --delete 1.6.4-0ubuntu1
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/aurora --delete 1.6.6-0ubuntu1
done

I've had a look at cairo-dock and cairo-dock-plug-ins branches as requested. Unfortunately the current branches have been extensively committed to in ways that the automatic importer cannot process, principally involving not maintaining a clean separation between upstream and packaging revisions. It is not obvious what should be done to fix them. I think we might need to discard the manually committed revisions.

@ Max Bowsher: thank you for your help!
It's not a problem to loose the history If it's the easiest solution.
I'm sorry, I didn't know that I had to use 'bzr merge-upstream' and not uscan and other commands as before (but there was no warning...)

gnu-fdisk fixup:

for d in lucid maverick natty oneiric; do
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/gnu-fdisk \
   -r revid:<email address hidden> upstream-1.2.3
done

laptop-mode-tools fixup:

bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/lucid/laptop-mode-tools -r revid:<email address hidden> upstream-1.52

linaro-image-tools fixup:

bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/natty/linaro-image-tools \
   -r revid:<email address hidden> upstream-0.4.3

elilo fixup:

for d in lucid maverick natty oneiric; do
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/elilo \
  -r <email address hidden> \
  upstream-3.10
done

python-django-openid-auth fixup:

for d in lucid maverick natty oneiric; do
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/python-django-openid-auth \
  -r revid:<email address hidden> \
  upstream-0.2
done

python-ldap fixup:

for d in lucid maverick natty oneiric; do
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/python-ldap \
  -r revid:<email address hidden> \
  upstream-2.3.11
done

python-virtualenv fixup:

bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/lucid/python-virtualenv -r revid:<email address hidden> upstream-1.4.5

refit fixup:

for d in lucid maverick natty oneiric; do
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/refit \
  -r revid:<email address hidden> \
  upstream-0.14
done

ruby-gnome2 fixup:

for d in lucid maverick natty oneiric; do
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/ruby-gnome2 \
  -r revid:<email address hidden> \
  upstream-0.19.3
done

yate fixup:

for d in maverick natty oneiric; do
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/yate \
  -r revid:<email address hidden> \
  upstream-2.2.0-1~dfsg
done

pygtksourceview fixup:

This is like the problem with the aurora package earlier - there is a spurious commit claiming to be a new upstream version, but which actually only updates debian/changelog. We need to remove that, and its associated tag:

for d in lucid maverick natty oneiric; do
bzr uncommit -r tag:2.9.2-0ubuntu2 --force \
  bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/pygtksourceview
bzr tag -d ubuntu-$d --delete 2.10.0-0ubuntu1
done

gpredict fixup:

for d in lucid maverick natty oneiric; do
bzr tag -d bzr+ssh://bazaar.launchpad.net/ubuntu/$d/gpredict \
  -r <email address hidden> \
  upstream-1.1
done

foo2zjs fixup:

for d in lucid maverick natty oneiric; do
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/foo2zjs \
  -r revid:<email address hidden> \
  upstream-20090908dfsg
done

Fixups applied for: gnu-fdisk, laptop-mode-tools, linaro-image-tools, elilo, python-django-openid-auth, python-ldap, python-virtualenv, refit, ruby-gnome2, yate, pygtksourceview.

Fixups applied for gpredict and foo2zjs too. (And all requeued as well of course.)

The above series of fixups completes, with the exception of qbzr which is held back pending another issue (incorrect selection of upstream-* tag as tip of upstream branch causes unnecessarily complex history), the fixups of all NoSuchTag failures caused by a past bug involving failure to copy tags from the Debian import branches.

The remaining instances of this particular failure signature mainly fall into the categories of:

 * people using merge-upstream, but importing the wrong versions
 * people not using merge-upstream
 * branches where instead of using merge-upstream, the upstream VCS is being directly merged into the packaging branch, so there are no upstream-* tags at all

Sorry but what can we do for cairo-dock branches? lp:ubuntu/cairo-dock, lp:ubuntu/natty/cairo-dock, lp:ubuntu/cairo-dock-plug-ins, lp:ubuntu/natty/cairo-dock-plug-ins ?
It seems a few upstream-* tags are missing because 'bzr merge-upstream' has not been used and I guess it's maybe easier to uncommit a few revisions but how many? I guess the rev 23 on lp:ubuntu/cairo-dock is incorrect but what about the older? Can you help me to fix these broken branch?

On Thu, 19 May 2011 12:46:25 -0000, Matthieu Baerts <email address hidden> wrote:
> Sorry but what can we do for cairo-dock branches? lp:ubuntu/cairo-dock, lp:ubuntu/natty/cairo-dock, lp:ubuntu/cairo-dock-plug-ins, lp:ubuntu/natty/cairo-dock-plug-ins ?
> It seems a few upstream-* tags are missing because 'bzr merge-upstream' has not been used and I guess it's maybe easier to uncommit a few revisions but how many? I guess the rev 23 on lp:ubuntu/cairo-dock is incorrect but what about the older? Can you help me to fix these broken branch?

Hi,

I've just tried to uncommit those revisions from the branch so that we
can get back to a known-good state, but something odd is going on.

Simply running bzr uncommit lp:ubuntu/maverick/cairo-dock gives a
backtrace complaining of NoSuchRevision, and I'm not sure why. Could
someone from the bzr team take a look please?

Thanks,

James

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 5/19/2011 8:35 PM, James Westby wrote:
> On Thu, 19 May 2011 12:46:25 -0000, Matthieu Baerts <email address hidden> wrote:
>> Sorry but what can we do for cairo-dock branches? lp:ubuntu/cairo-dock, lp:ubuntu/natty/cairo-dock, lp:ubuntu/cairo-dock-plug-ins, lp:ubuntu/natty/cairo-dock-plug-ins ?
>> It seems a few upstream-* tags are missing because 'bzr merge-upstream' has not been used and I guess it's maybe easier to uncommit a few revisions but how many? I guess the rev 23 on lp:ubuntu/cairo-dock is incorrect but what about the older? Can you help me to fix these broken branch?
>
> Hi,
>
> I've just tried to uncommit those revisions from the branch so that we
> can get back to a known-good state, but something odd is going on.
>
> Simply running bzr uncommit lp:ubuntu/maverick/cairo-dock gives a
> backtrace complaining of NoSuchRevision, and I'm not sure why. Could
> someone from the bzr team take a look please?
>
> Thanks,
>
> James
>

I'm pretty sure this is: bug #785116

It wouldn't surprise me too much if lp:ubuntu/* branches ended up as
stacked branches with absolutely 0 local revisions (vs the dev focus).

One option would be to use "bzr push --overwrite -r -2" or something
like that. "uncommit" fails because it is trying to show the log of the
last commit.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk3Vbu8ACgkQJdeBCYSNAAP3LQCdHha/VvC4DyxZdQDaf7p1U9MN
sKkAmQG+2vckWFzLNVemGjgXiFQhDlrl
=iXwD
-----END PGP SIGNATURE-----

John A Meinel wrote:
[…]
> > Simply running bzr uncommit lp:ubuntu/maverick/cairo-dock gives a
> > backtrace complaining of NoSuchRevision, and I'm not sure why. Could
> > someone from the bzr team take a look please?
[…]
>
> I'm pretty sure this is: bug #785116
>
> It wouldn't surprise me too much if lp:ubuntu/* branches ended up as
> stacked branches with absolutely 0 local revisions (vs the dev focus).

Yep, exactly. Another workaround, as mentioned on the bug, is to use
nosmart+ or sftp:// URLs to avoid the faulty HPSS logic.

-Andrew.

OK, so for cairo-dock, I propose that we:

# Back up the existing branches

bzr branch --no-tree bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/lucid/cairo-dock cairo-dock-tmp
cd cairo-doc-tmp
bzr push bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/cairo-dock/lucid-`date +%Y%m%d%H%M`
bzr pull bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/maverick/cairo-dock
bzr push bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/ubuntu/maverick/cairo-dock/maverick-`date +%Y%m%d%H%M`
cd ..
rm -r cairo-dock-tmp

# (There are no extra revisions in natty / oneiric beyond those in the maverick branch)

# Now pop off the manual commits that mix upstream and packaging

for d in lucid maverick natty oneiric; do
bzr uncommit -r tag:2.0.9-0ubuntu1 nosmart+bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/cairo-dock
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/cairo-dock --delete 2.1.3-10-lucid-0ubuntu1
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/cairo-dock --delete 2.2.0~4-0ubuntu1
done

And for cairo-dock-plug-ins, I propose that we:

# Back up the existing branches

bzr branch --no-tree bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/lucid/cairo-dock-plug-ins cairo-dock-plug-ins-tmp
cd cairo-doc-plug-ins-tmp
bzr push bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/ubuntu/lucid/cairo-dock-plug-ins/lucid-`date +%Y%m%d%H%M`
bzr pull bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/maverick/cairo-dock-plug-ins
bzr push bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/ubuntu/maverick/cairo-dock-plug-ins/maverick-`date +%Y%m%d%H%M`
cd ..
rm -r cairo-dock-plug-ins-tmp

# (There are no extra revisions in natty / oneiric beyond those in the maverick branch)

# Now pop off the manual commits that mix upstream and packaging

for d in lucid maverick natty oneiric; do
bzr uncommit -r tag:2.1.3-6-0ubuntu1 nosmart+bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/cairo-dock-plug-ins
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/cairo-dock-plug-ins --delete 2.1.3-10-lucid-0ubuntu1
bzr tag -d bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/$d/cairo-dock-plug-ins --delete 2.1.3-10-lucid-0ubuntu2
done

I've applied those adjustments to cairo-dock and cairo-dock-plug-ins, and requeued them.

@ Max & Andrew: thank you for your help! :)

At this point in time, 14+3 (there are two different failure signatures) packages remain affected. Here are my notes on these:

qbzr: This remains affected by fallout from the previous bug causing tags not to be merged from Debian. We have not fixed it up yet, because local testing has determined that the resulting import would be affected by another bug (picking the wrong upstream-* tag to extract as the upstream branch) producing contorted history.

clipit, gm-notify: Each contains only two package versions, the second in each branch committed by a human, with incorrect specification of the entire package version as the upstream version. My recommendation is to delete the branches from Launchpad entirely and redo the import cleanly.

python-configglue: Fairly similar to the above, there was an upstream version imported with the incorrect version number. As the branches include only 4 package versions across two upstream versions, and little interesting history, I recommend deleting these branches and redoing the import cleanly.

apport: A somewhat muddled awkward situation has arisen here, because, I believe, existing manually maintained branches have been designated as official package branches without fully considering the consequences for the importer and for the branch-distro process used when a new Ubuntu series is created.
Specifically, there exist lp:~ubuntu-core-dev/ubuntu/$series/apport/ubuntu branches for hardy, karmic, lucid, maverick, natty and oneiric, and this is where the actual development takes place. Of these, the maverick branch *only* is marked as an official branch. However, the branch-distro process appears to have created official branches for natty and oneiric at lp:~ubuntu-core-dev/ubuntu/$series/apport/$series - note the differing last component of $series vs. ubuntu. The human packagers have ignored these autocreated branches and created their own following their pre-existing naming convention.
In the short term, we should delete the spurious unmaintained natty and oneiric branches, shifting the official branch links to the branches that are actually being used. In the long-term, we need to do something to stop the situation re-occurring, but I'm not sure what.
The ~ubuntu-core-dev/..../ubuntu branches are in active use and are derived from the project's upstream VCS. As a result, the upstream version tags are just "VERSION" not "upstream-VERSION", and the importer does not understand this well.

langpack-locales: This branch is in active use for versioning the actual packages uploaded to the archive, but has had numerous new upstream versions manually committed directly to the packaging branch.

openssh, upstart, wncksync: This branch is in active use and appears to be constructed using a direct import of the upstream VCS. As a result, the upstream version tags are just "VERSION" not "upstream-VERSION", and the importer does not understand this well.

clutk: The karmic branch is importer-created. The lucid, maverick, natty and oneiric branches all have identical content, and are derived from upstream VCS. Unfortunately, they don't match what's actually packaged in the archive in any of these releases :-(. Therefo...

We're now down to just 7 packages, qbzr apport clutk langpack-locales openssh upstart wncksync.

The Upstart issue is tracked on bug 813937.

This bug is currently being reported as an issue for facter, hamster-applet, and qbzr at http://package-import.ubuntu.com/status/. I arrived there after the packaging branch for qbzr was being reported as out-of-date when branching it in Ubuntu.

I wanted to checkout pidgin using command `bzr branch lp:ubuntu/trusty/pidgin`, however that fails with the following log:
```
Most recent Ubuntu Wily version: 1:2.10.11-0ubuntu4
Packaging branch version: 1:2.10.3-0ubuntu1
Packaging branch status: OUT-OF-DATE
Most recent Ubuntu Vivid version: 1:2.10.9-0ubuntu8
Packaging branch version: 1:2.10.3-0ubuntu1
Packaging branch status: OUT-OF-DATE
Most recent Ubuntu Utopic version: 1:2.10.9-0ubuntu7
Packaging branch version: 1:2.10.3-0ubuntu1
Packaging branch status: OUT-OF-DATE
Most recent Ubuntu Trusty version: 1:2.10.9-0ubuntu3
Packaging branch version: 1:2.10.3-0ubuntu1
Packaging branch status: OUT-OF-DATE
bzr: ERROR: Revision {<email address hidden>} not present in "Graph(StackedParentsProvider(bzrlib.repository._LazyListJoin(([CachingParentsProvider(CallableToParentsProviderAdapter(<bound method CHKInventoryRepository._get_parent_map_no_fallbacks of CHKInventoryRepository('http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/pidgin/trusty/.bzr/repository/')>))], []))))".
```

The importer <http://package-import.ubuntu.com/status/pidgin.html#2012-07-04 17:31:07.840197> pointed me to this issue. How do I clone pidgin package source? I want to add a patch to the package.