Launchpad code imports

Mirror of charm-nova-compute is failing due to "strange" timeout

Bug #1961804 reported by Alex Kavanagh
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad code imports
Fix Released
High
Colin Watson

Bug Description

The charm-nova-compute project holds the code upstream at opendev.org. The code mirror is failing.

Code import page: https://code.launchpad.net/~openstack-charmers/charm-nova-compute/+git/charm-nova-compute

Error from log:

2022-02-22 12:55:13 INFO Starting job.
2022-02-22 12:55:13 INFO Getting existing repository from hosting service.
2022-02-22 12:55:19 INFO Cloning into bare repository 'repository'...
2022-02-22 12:55:19 INFO remote: Enumerating objects: 27119, done.
2022-02-22 12:55:19 INFO remote: Counting objects: 100% (27119/27119)
2022-02-22 12:55:19 INFO remote: Counting objects: 100% (27119/27119), done.
2022-02-22 12:55:21 INFO remote: Compressing objects: 100% (11644/11644)
2022-02-22 12:55:21 INFO remote: Compressing objects: 100% (11644/11644), done.
2022-02-22 12:55:21 INFO Receiving objects: 98% (26577/27119), 14.57 MiB | 14.42 MiB/s
2022-02-22 12:55:21 INFO remote: Total 27119 (delta 16249), reused 25674 (delta 14827)
2022-02-22 12:55:21 INFO Receiving objects: 100% (27119/27119), 14.57 MiB | 14.42 MiB/s
2022-02-22 12:55:21 INFO Receiving objects: 100% (27119/27119), 17.43 MiB | 14.42 MiB/s, done.
2022-02-22 12:55:22 INFO Resolving deltas: 100% (16249/16249)
2022-02-22 12:55:22 INFO Resolving deltas: 100% (16249/16249), done.
2022-02-22 12:55:22 INFO Checking connectivity... done.
2022-02-22 12:55:22 INFO Fetching remote repository.
Traceback (most recent call last):
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/scripts/code-import-worker.py", line 112, in <module>
    sys.exit(script.main())
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/scripts/code-import-worker.py", line 107, in main
    return import_worker.run()
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/lib/lp/codehosting/codeimport/worker.py", line 576, in run
    return self._doImport()
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/lib/lp/codehosting/codeimport/worker.py", line 1188, in _doImport
    cwd="repository")
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/lib/lp/codehosting/codeimport/worker.py", line 1070, in _runGit
    for line in self._throttleProgress(git_process.stdout):
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/lib/lp/codehosting/codeimport/worker.py", line 1027, in _throttleProgress
    buffered, timeout=timeout):
  File "/srv/lp-codeimport/payloads/0f80d22c76a72f9e9eb763eb46f19ce67bb48415/lib/lp/codehosting/codeimport/worker.py", line 1046, in _throttleProgress
    line = next(wrapped_file)
KeyboardInterrupt
Import failed:
Traceback (most recent call last):
Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure.

Related branches

~cjwatson/lp-codeimport:git-auth-launchpad-services
Ioana Lasc (community): Approve
Diff: 22 lines (+2/-3)
1 file modified
lib/lp/codehosting/codeimport/worker.py (+2/-3)
~cjwatson/lp-codeimport:git-fetch-progress
Jürgen Gmach: Approve
Diff: 13 lines (+1/-1)
1 file modified
lib/lp/codehosting/codeimport/worker.py (+1/-1)
Revision history for this message
Colin Watson (cjwatson) wrote :

After hunting down a few red herrings, I'm currently baffled by this. My best theory is to add some more progress information to the `git fetch` that it's running at that point and try to see what's going on.

Changed in lp-codeimport:
importance: Undecided → High
status: New → Triaged
Revision history for this message
Colin Watson (cjwatson) wrote :

Adding progress information didn't help. I found a way to reproduce it reasonably reliably using just git, and bisected this to https://github.com/git/git/commit/eb049759fb6b739310af52ee0e13ce6cd0c86be7 which fixes the misbehaviour. (Even after that, the tip of git main still exhibits a similar hang if I force the use of protocol v0.)

However, this is definitely not something that can simply be cherry-picked to xenial's git, since none of the v2 stuff existed there. I think we may have to backport jammy's git to xenial.

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

It's probably not an appropriate upgrade, but may be a useful starting point, but there is the git core ppa with the latest version for xenial: https://launchpad.net/~git-core/+archive/ubuntu/ppa/+packages?field.name_filter=&field.status_filter=published&field.series_filter=xenial

Revision history for this message
Colin Watson (cjwatson) wrote :

I've got suitable packaging already now - the current blocker is just making sure it works across our services, since we don't like to get into a situation where we're using entirely different backports on different services (though a certain amount of divergence is inevitable due to running different Ubuntu releases).

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Hi

Is there any progress on this, please? This is rapidly becoming critical as we are nearing release and can't build the charms in launchpad.

Colin Watson (cjwatson)
Changed in lp-codeimport:
status: Triaged → In Progress
assignee: nobody → Colin Watson (cjwatson)
status: In Progress → Fix Committed
Revision history for this message
Colin Watson (cjwatson) wrote :

I've deployed a fix to lp-codeimport itself to allow it to *tolerate* new versions of git (upgrading git would previously have caused failures to push imported repositories back to Launchpad).

Following up to that, I've filed a ticket with our sysadmins to run a charm upgrade and to upgrade to the new git version in our PPA. I don't have the ticket number yet since some bit of email delivery somewhere seems to be slower than usual, but once that ticket is resolved it should fix this.

Revision history for this message
Colin Watson (cjwatson) wrote :

https://portal.admin.canonical.com/C149076

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Thanks very much; I'll keep an eye out for the progress of the ticket and then verify the code imports.

Revision history for this message
Colin Watson (cjwatson) wrote :

This should all be sorted now, and I've retried the code imports of charm-nova-compute, charm-ceph-mon, and charm-keystone which were the only openstack-charmers failures from this that I could find.

Changed in lp-codeimport:
status: Fix Committed → Fix Released
summary: - Mirror of nova-compute is failing due to "strange" timeout
+ Mirror of charm-nova-compute is failing due to "strange" timeout
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.