apt-transport-https: large packages timeout after 120s, even while progress is being made

Bug #354972 reported by Steve Beattie
22
This bug affects 2 people
Affects Status Importance Assigned to Milestone
apt (Debian)
Fix Released
Unknown
apt (Ubuntu)
Fix Released
High
Michael Vogt
Karmic
Won't Fix
Undecided
Unassigned
Lucid
Fix Released
High
Michael Vogt

Bug Description

Binary package hint: apt

As part of testing, I am occasionally installing large packages from private-ppas on launchpad, which makes use of https and apt-transport-https. While doing so, I get timeouts like the following:

  Get:1 https://private-ppa.launchpad.net hardy/main PACKAGE 9.1.0-6 [63.4MB]
  Err https://private-ppa.launchpad.net hardy/main PACKAGE 9.1.0-6
    Operation timed out after 120000 milliseconds with 13942784 out of 63370760 bytes received
  Failed to fetch https://ID:<email address hidden>/PROJECT/ubuntu/pool/main/P/PACKAGE/PACKAGE_9.1.0-6_i386.deb Operation timed out after 120000 milliseconds with 13942784 out of 63370760 bytes received
  E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Manually wget'ing the package and installing succeeds, and I also do not see similar timeouts when downloading (asbsurdly) large packages over http.

Looking at the source code, in methods/https.cc we have the following code:

   // set timeout
   int timeout = _config->FindI("Acquire::http::Timeout",120);
   curl_easy_setopt(curl, CURLOPT_TIMEOUT, timeout);
   curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, timeout);

Quoting the curl_easy_setopt(3) manpage:

       CURLOPT_TIMEOUT
              Pass a long as parameter containing the maximum time in seconds
              that you allow the libcurl transfer operation to take.

Note that it's not a timeout of inactivity, it's just a maximum time the transfer can take. This does not match the behavior of method/http.cc where HttpMethod::Go sets the same timeout value on a select() call on any of the open filedescriptors; this only gets triggered if none of the connections make any progress by the timeout time.

It seems to me that the http and https behaviors ought to be made consistent, or else the https method ought to honor a separate Acquire::https::Timeout setting.

ProblemType: Bug
Architecture: i386
DistroRelease: Ubuntu 9.04
Package: apt-transport-https 0.7.20.2ubuntu5
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: apt
Uname: Linux 2.6.28-11-generic i686

Revision history for this message
Steve Beattie (sbeattie) wrote :
Changed in apt (Debian):
status: Unknown → Incomplete
Revision history for this message
Michael Casadevall (mcasadevall) wrote :

Ran into this bug at UDS. Caused me quite a bit of trouble; it seems the issue is that http and https work differently with the same options applied based off the Debian bug.

Changed in apt (Ubuntu):
importance: Undecided → High
milestone: none → karmic-alpha-3
status: New → Confirmed
tags: added: oem-services
Revision history for this message
Javier Collado (javier.collado) wrote :

Hello,

I've also had the same problem so I entered the same apt-get install command several times so as to get the package completely downloaded. Unfortunately, when the download was expected to finish, it failed and, on calling once more the apt-get install command, it started from the begining again. Is there any workaround for this?

Best regards,
   Javier

Colin Watson (cjwatson)
Changed in apt (Ubuntu):
assignee: nobody → Michael Vogt (mvo)
Revision history for this message
Brian Thomason (brian-thomason) wrote :

It appears CURLOPT_TIMEOUT isn't proper to use here, as indeed it times out in 120s regardless of the connection state. The fact the connection doesn't resume seems to be another issue, but I have attached a small patch that fixes the timeout issue. It sets the minimum transfer speed to 1bs, and uses the timeout of 120s only if < 1bs is being transferred.

Revision history for this message
aman (joseph6565) wrote :

I am also facing the same issue.
Can somebody help me in resolving this.
What is the path of https.cc & how to put the changes suggested by Brian Thomason in his comment.

Revision history for this message
Roderick B. Greening (roderick-greening) wrote :

This bug is still present under Karmic. This prevents updates for Kubuntu-ppa testing from working without manual intervention.

Michael Vogt (mvo)
Changed in apt (Ubuntu Lucid):
milestone: karmic-alpha-3 → lucid-alpha-1
status: Confirmed → Fix Committed
Changed in apt (Debian):
status: Incomplete → Fix Committed
Michael Vogt (mvo)
Changed in apt (Ubuntu Lucid):
milestone: lucid-alpha-1 → lucid-alpha-2
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.9 KiB)

This bug was fixed in the package apt - 0.7.24ubuntu1

---------------
apt (0.7.24ubuntu1) lucid; urgency=low

  [ Michael Vogt ]
  * apt-pkg/deb/dpkgpm.cc:
    - include df -l output in the apport log as well (thanks to
      tjaalton)
  * apt-pkg/packagemanager.cc:
    - add output about pre-depends configuring when debug::pkgPackageManager
      is used
  * methods/https.cc:
    - fix incorrect use of CURLOPT_TIMEOUT, closes: #497983, LP: #354972
      thanks to Brian Thomason for the patch
  * merge lp:~mvo/apt/netrc branch, this adds support for a
    /etc/apt/auth.conf that can be used to store username/passwords
    in a "netrc" style file (with the extension that it supports "/"
    in a machine definition). Based on the maemo git branch.

  [ Brian Murray ]
  * apt-pkg/depcache.cc, apt-pkg/indexcopy.cc:
    - typo fix (LP: #462328)

apt (0.7.24) unstable; urgency=low

  [ Nicolas François ]
  * Cleaned up the first patch draft from KURASAWA Nozomu to finally
    get po4a support for translating the man pages.
    Many thanks to both for this excellent work! (Closes: #441608)
  * doc/ja/*, doc/po/ja.po:
    - remove the old ja man page translation and replace it with
      the new po4a-powered translation by KURASAWA Nozomu.
  * doc/*.?.xml (manpages):
    - add contrib to author tags and also add refmiscinfo to fix warnings
  * doc/style.txt, buildlib/defaults.mak, buildlib/manpage.mak:
    - fix a few typos in the comments of this files

  [ Michael Vogt ]
  * apt-pkg/deb/dpkgpm.cc:
    - when tcgetattr() returns non-zero skip all pty magic
      (thanks to Simon Richter, closes: #509866)
  * apt-inst/contrib/arfile.cc:
    - show propper error message for Invalid archive members

  [ David Kalnischkies ]
  * doc/Doxyfile.in:
    - update file with doxygen 1.6.1 (current unstable)
    - activate DOT_MULTI_TARGETS, it is default on since doxygen 1.5.9
  * buildlib/po4a_manpage.mak, doc/makefile, configure:
    - simplify the makefiles needed for po4a manpages
  * apt-pkg/contrib/configuration.cc:
    - add a helper to easily get a vector of strings from the config
  * apt-pkg/contrib/strutl.cc:
    - replace unknown multibytes with ? in UTF8ToCharset (Closes: #545208)
  * doc/apt-get.8.xml:
    - fix two little typos in the --simulate description. (Closes: #545059)
  * apt-pkg/aptconfiguration.cc, doc/apt.conf.5.xml:
    - add an order subgroup to the compression types to simplify reordering
      a bit and improve the documentation for this option group.
  * doc/apt.conf.5.xml:
    - document the Acquire::http::Dl-Limit option
    - try to be crystal clear about the usage of :: and {} (Closes: #503481)
  * doc/apt-cache.8.xml:
    - clarify the note for the pkgnames command (Closes: #547599)
  * doc/apt.ent, all man pages:
    - move the description of files to globally usable entities
  * doc/apt_preferences.5.xml:
    - document the new preferences.d folder (Closes: #544017)
  * methods/rred.cc:
    - add at the top without failing (by Bernhard R. Link, Closes: #545694)
  * buildlib/sizetable:
    - add amd64 for cross building (by Mikhail Gusarov, Closes: #513058)
  * debian/prerm:
    - remove file as nobody will upgrade from 0....

Read more...

Changed in apt (Ubuntu Lucid):
status: Fix Committed → Fix Released
Changed in apt (Debian):
status: Fix Committed → Fix Released
Revision history for this message
Alessandro Ghersi (alessandro-ghersi) wrote :

I still have this problem on Lucid with apt 0.7.25ubuntu2

Get:1 https://private-ppa.launchpad.net lucid/main python-qt4-dbg 4.7.0-0ubuntu1~ppa1 [27.0MB]
Err https://private-ppa.launchpad.net lucid/main python-qt4-dbg 4.7.0-0ubuntu1~ppa1
  Operation timed out after 120000 milliseconds with 19349504 out of 26997332 bytes received
E: Failed to fetch https://alessandro-ghersi....................................................
timed out after 120000 milliseconds with 19349504 out of 26997332 bytes received

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt (Ubuntu Karmic):
status: New → Confirmed
Revision history for this message
martin suchanek (martin-suc) wrote :

I have the same problem

ldconfig deferred processing now taking place
Processing triggers for menu ...
Error: Timeout was reached

following:
echo 'Acquire::http::Timeout "600";' > timeout

did not help.

Is there any workaround to fix it ?

thank you.

Revision history for this message
Rolf Leggewie (r0lf) wrote :

karmic has seen the end of its life and is no longer receiving any updates. Marking the karmic task for this ticket as "Won't Fix".

Changed in apt (Ubuntu Karmic):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.