Bug #1003633 “Escape plus character in apt HTTP requests to work...” : Bugs : apt package : Ubuntu

Revision history for this message

Daniel Hartwig (wigs) wrote on 2012-05-24: Re: [Bug 1003633] [NEW] Escape plus character in apt HTTP requests to work around Amazon S3 bug

#1

Also, this 2007 thread:

[1] https://forums.aws.amazon.com/thread.jspa?threadID=16361

where AWS representatives expound on their "justification" some more.
Instead of being well-behaved netizens, they just document their
non-conformance – though there is no indication in these threads where
such documentation is, other than provided in examples.

That thread also indicates that the path is expected to be
URL-encoded, not just some characters. This probably means including
':' and others to avoid more unanticipated transforms.

Revision history for this message

Daniel Hartwig (wigs) wrote on 2012-05-24:

#2

Effectively S3 does not implement HTTP according to the spec. Rather
than working around the behaviour of S3 mirrors in methods/http we
could define methods/s3. S3 mirrors would then be properly identified
as non-http in sources.list.

This method could be a wrapper which invokes the http method with a
transformed URI.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2012-05-30:

#3

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt (Ubuntu):
status:	New → Confirmed

Revision history for this message

Julian Andres Klode (juliank) wrote on 2012-05-30:

#4

Can't they just encode the plus character as a space in their S3 instances? Then a file like package_1.0+really0.9_all.deb would be stored as "package_1.0 really0.9_all.deb" and the URI decoding done by S3 is circumvented server-side, instead of requiring hacks on the client side.

Revision history for this message

Daniel Hartwig (wigs) wrote on 2012-06-14:

#5

> Can't they just encode the plus character as a space in their S3 instances?

That does seem preferable.

Is there a package that is commonly used to maintain a mirror on S3 where this change can be made?

Revision history for this message

Michael Vogt (mvo) wrote on 2013-01-11:

#6

Trivial extension of Daniels fix for #1086997 Edit (542 bytes, text/plain)

Ubuntu Foundations Team Bug Bot (crichton) on 2013-01-11

tags:

added: patch

Revision history for this message

Daniel Hartwig (wigs) wrote on 2013-01-12: Re: [Bug 1003633] Re: Escape plus character in apt HTTP requests to work around Amazon S3 bug

#7

On 11 January 2013 16:58, Michael Vogt <email address hidden> wrote:
> ** Patch added: "Trivial extension of Daniels fix for #1086997"
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1003633/+attachment/3479965/+files/fix.diff

Please include also a comment above that line indicating why “+” will
be encoded, lest a future optimizer decides to remove it.

Michael Vogt (mvo) on 2013-01-24

Changed in apt (Ubuntu):
status:	Confirmed → Fix Committed

Revision history for this message

Darik Horn (dajhorn) wrote on 2013-01-27:

#8

This non-conformance bug works in both directions, so an easy kludge is to hard link all file names like this:

$ ln foobar-1.2+baz3_all.deb 'foobar-1.2 baz3_all.deb'

And sync to the s3 bucket normally. Afterwards, the key can be called through http with a literal "+" character or the "%2B" escape. Most clients like `apt-get` and `wget` will write out a file with the correct name using either form in the URL.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2013-07-10:

#9

Download full text (13.3 KiB)

This bug was fixed in the package apt - 0.9.9.1~ubuntu1

---------------
apt (0.9.9.1~ubuntu1) saucy; urgency=low

  * merged from the debian/sid branch:
    - debian/gbp.conf: change build branch to ubuntu/master
    - use ubuntu keyring and ubuntu archive keyring in apt-key
    - run update-apt-xapian-index in apt.cron
    - run apt-key net-update in cron.daily
    - different example sources.list
    - APT::pkgPackageManager::MaxLoopCount set to 5000
    - apport pkgfailure handling
    - ubuntu changelog download handling
    - patch for apt cross-building, see http://bugs.debian.org/666772
    - debian/apt.auto-removal.sh
      + make kernels auto-removable

apt (0.9.9.1) UNRELEASED; urgency=low

* debian/rules:
- call dh_clean in clean (closes: #714980)

apt (0.9.9) unstable; urgency=low

  [ Michael Vogt ]
  * improve debug output for the Debug::pkgProblemResolver and
    Debug::pkgDepCache::AutoInstall
  * improve apt-cdrom output when no CD-ROM can be auto-detected
  * document --no-auto-detect in apt-cdrom

  [ David Kalnischkies ]
  * build the en manpages in subdirectory doc/en
  * remove -ldl from cdrom and -lutil from apt-get linkage
  * rewrite pkgOrderList::DepRemove to stop incorrect immediate setting
    (Closes: 645713)
  * prefer Essentials over Removals in ordering score
  * fix priority sorting by prefering higher in MarkInstall
  * try all providers in order if uninstallable in MarkInstall
  * do unpacks before configures in SmartConfigure (Closes: #707578)
  * fix support for multiple patterns in apt-cache search (Closes: #691453)
  * set Fail flag in FileFd on all errors consistently
  * don't explicitly init ExtractTar InFd with invalid fd
  * OpenDescriptor should autoclose fd always on error (Closes: #704608)
  * fail in CopyFile if the FileFds have error flag set
  * ensure state-dir exists before coyping cdrom files
  * fix file location for configure-index.gz in apt.conf(5) (Closes: #711921)
  * handle missing "Description" in apt-cache show (Closes: #712435)
  * try defaults if auto-detection failed in apt-cdrom (Closes: #712433)
  * support \n and \r\n line endings in ReadMessages
  * do not redownload unchanged InRelease files
  * trigger NODATA error for invalid InRelease files (Closes: #712486)

apt (0.9.8.2) unstable; urgency=low

[ Programs translations ]
* French translation : typo fix. Closes: #677272

[ Guillem Jover ]
* Update Vcs fields (Closes: #708562)

  [ Michael Vogt ]
  * buildlib/apti18n.h.in:
    - fix build failure when building without NLS (closes: #671587)

[ Gregoire Menuel ]
* Fix double free (closes: #711045)

  [ Raphael Geissert ]
  * Fix crash when the "mirror" method does not find any entry
    (closes: #699303)

  [ Johan Kiviniemi ]
  * cmdline/apt-key:
    - Create new keyrings with mode 0644 instead of 0600.
    - Accept a nonexistent --keyring file with the adv subcommand as well.

apt (0.9.8.1) unstable; urgency=low

  [ David Kalnischkies ]
  * apt-pkg/indexcopy.cc:
    - non-inline RunGPGV methods to restore ABI compatibility with previous
      versions to fix partial upgrades (Closes: #707771)

[ Michael Vogt ]
* moved source to http://git.debian.org/...

This bug was fixed in the package apt - 0.9.9.1~ubuntu1

---------------
apt (0.9.9.1~ubuntu1) saucy; urgency=low

* merged from the debian/sid branch:
    - debian/gbp.conf: change build branch to ubuntu/master
    - use ubuntu keyring and ubuntu archive keyring in apt-key
    - run update-apt-xapian-index in apt.cron
    - run apt-key net-update in cron.daily
    - different example sources.list
    - APT::pkgPackageManager::MaxLoopCount set to 5000
    - apport pkgfailure handling
    - ubuntu changelog download handling
    - patch for apt cross-building, see http://bugs.debian.org/666772
    - debian/apt.auto-removal.sh
      + make kernels auto-removable

apt (0.9.9.1) UNRELEASED; urgency=low

* debian/rules:
    - call dh_clean in clean (closes: #714980)

apt (0.9.9) unstable; urgency=low

[ Michael Vogt ]
  * improve debug output for the Debug::pkgProblemResolver and
    Debug::pkgDepCache::AutoInstall
  * improve apt-cdrom output when no CD-ROM can be auto-detected
  * document --no-auto-detect in apt-cdrom

[ David Kalnischkies ]
  * build the en manpages in subdirectory doc/en
  * remove -ldl from cdrom and -lutil from apt-get linkage
  * rewrite pkgOrderList::DepRemove to stop incorrect immediate setting
    (Closes: 645713)
  * prefer Essentials over Removals in ordering score
  * fix priority sorting by prefering higher in MarkInstall
  * try all providers in order if uninstallable in MarkInstall
  * do unpacks before configures in SmartConfigure (Closes: #707578)
  * fix support for multiple patterns in apt-cache search (Closes: #691453)
  * set Fail flag in FileFd on all errors consistently
  * don't explicitly init ExtractTar InFd with invalid fd
  * OpenDescriptor should autoclose fd always on error (Closes: #704608)
  * fail in CopyFile if the FileFds have error flag set
  * ensure state-dir exists before coyping cdrom files
  * fix file location for configure-index.gz in apt.conf(5) (Closes: #711921)
  * handle missing "Description" in apt-cache show (Closes: #712435)
  * try defaults if auto-detection failed in apt-cdrom (Closes: #712433)
  * support \n and \r\n line endings in ReadMessages
  * do not redownload unchanged InRelease files
  * trigger NODATA error for invalid InRelease files (Closes: #712486)

apt (0.9.8.2) unstable; urgency=low

[ Programs translations ]
  * French translation : typo fix. Closes: #677272

[ Guillem Jover ]
  * Update Vcs fields (Closes: #708562)

[ Michael Vogt ]
  * buildlib/apti18n.h.in:
    - fix build failure when building without NLS (closes: #671587)

[ Gregoire Menuel ]
  * Fix double free (closes: #711045)

[ Raphael Geissert ]
  * Fix crash when the "mirror" method does not find any entry
    (closes: #699303)

[ Johan Kiviniemi ]
  * cmdline/apt-key:
    - Create new keyrings with mode 0644 instead of 0600.
    - Accept a nonexistent --keyring file with the adv subcommand as well.

apt (0.9.8.1) unstable; urgency=low

[ David Kalnischkies ]
  * apt-pkg/indexcopy.cc:
    - non-inline RunGPGV methods to restore ABI compatibility with previous
      versions to fix partial upgrades (Closes: #707771)

[ Michael Vogt ]
  * moved source to http://git.debian.org/apt/apt.git
  * updated gbp.conf to match what bzr-buildpackage is doing
  * remove .bzr-buildpackage/default.conf (superseeded by gbp.conf)

apt (0.9.8) unstable; urgency=low

[ Ludovico Cavedon ]
  * properly handle if-modfied-since with libcurl/https
    (closes: #705648)

[ Andreas Beckman ]
  * apt-pkg/algorithms.cc:
    - Do not propagate negative scores from rdepends. Propagating the absolute
      value of a negative score may boost obsolete packages and keep them
      installed instead of installing their successors.  (Closes: #699759)

[ Michael Vogt ]
  * apt-pkg/sourcelist.cc:
    - fix segfault when a hostname contains a [, thanks to
      Tzafrir Cohen (closes: #704653)
  * debian/control:
    - replace manpages-it (closes: #704723)

[ David Kalnischkies ]
  * various simple changes to fix cppcheck warnings
  * apt-pkg/pkgcachegen.cc:
    - do not store the MD5Sum for every description language variant as
      it will be the same for all so it can be shared to save cache space
    - handle language tags for descriptions are unique strings to be shared
    - factor version string creation out of NewDepends, so we can easily reuse
      version strings e.g. for implicit multi-arch dependencies
    - equal comparisions are used mostly in same-source relations,
      so use this to try to reuse some version strings
    - sort group and package names in the hashtable on insert
    - share version strings between same versions (of different architectures)
      to save some space and allow quick comparisions later on
  * apt-pkg/pkgcache.cc:
    - assume sorted hashtable entries for groups/packages
  * apt-pkg/cacheiterators.h:
    - provide DepIterator::IsSatisfied as a nicer shorthand for DepCheck
  * apt-pkg/deb/debversion.cc:
    - add a string-equal shortcut for equal version comparisions

[ Marc Deslauriers ]
  * make apt-ftparchive generate missing deb-src hashes (LP: #1078697)

[ Yaroslav Halchenko ]
  * Fix English spelling error in a message ('A error'). Unfuzzy
    translations. Closes: #705087

[ Programs translations ]
  * French translation completed (Christian Perrier)

[ Manpages translations ]
  * French translation completed (Christian Perrier)

[ Daniel Hartwig ]
  * apt-pkg/contrib/strutl.cc:
    - include port in shortened URIs (e.g. with apt-cache policy, progress
      display) thanks to James McCoy (Closes: #154868, #322074)
    - percent-encode username and password when writing URIs
  * methods/http.cc:
    - properly escape IP-literals (e.g. IPv6 address) when building
      Host headers and URIs (Closes: #620344)
  * methods/https.cc:
    - use https_proxy environment variable if present, falling back to
      http_proxy otherwise
    - use authentication credentials from proxy URI
      (Closes: #651640, LP: #1087512)
    - environment variables do not override an explicit no proxy
      directive ("DIRECT") in apt.conf
    - disregard all_proxy environment variable, like other methods

apt (0.9.7.9~exp3) experimental; urgency=low

[ Michael Vogt ]
  * apt-pkg/sourcelist.cc:
    - fix segfault when a hostname contains a [, thanks to
      Tzafrir Cohen (closes: #704653)
  * debian/control:
    - replace manpages-it (closes: #704723)

[ David Kalnischkies ]
  * various simple changes to fix cppcheck warnings
  * apt-pkg/pkgcachegen.cc:
    - do not store the MD5Sum for every description language variant as
      it will be the same for all so it can be shared to save cache space
    - handle language tags for descriptions are unique strings to be shared
    - factor version string creation out of NewDepends, so we can easily reuse
      version strings e.g. for implicit multi-arch dependencies
    - equal comparisions are used mostly in same-source relations,
      so use this to try to reuse some version strings
    - sort group and package names in the hashtable on insert
    - share version strings between same versions (of different architectures)
      to save some space and allow quick comparisions later on
  * apt-pkg/pkgcache.cc:
    - assume sorted hashtable entries for groups/packages
  * apt-pkg/cacheiterators.h:
    - provide DepIterator::IsSatisfied as a nicer shorthand for DepCheck
  * apt-pkg/deb/debversion.cc:
    - add a string-equal shortcut for equal version comparisions

[ Marc Deslauriers ]
  * make apt-ftparchive generate missing deb-src hashes (LP: #1078697)

[ Yaroslav Halchenko ]
  * Fix English spelling error in a message ('A error'). Unfuzzy
    translations. Closes: #705087

[ Programs translations ]
  * French translation completed (Christian Perrier)

[ Manpages translations ]
  * French translation completed (Christian Perrier)

[ Daniel Hartwig ]
  * apt-pkg/contrib/strutl.cc:
    - include port in shortened URIs (e.g. with apt-cache policy, progress
      display) thanks to James McCoy (Closes: #154868, #322074)
    - percent-encode username and password when writing URIs
  * methods/http.cc:
    - properly escape IP-literals (e.g. IPv6 address) when building
      Host headers and URIs (Closes: #620344)
  * methods/https.cc:
    - use https_proxy environment variable if present, falling back to
      http_proxy otherwise
    - use authentication credentials from proxy URI
      (Closes: #651640, LP: #1087512)
    - environment variables do not override an explicit no proxy
      directive ("DIRECT") in apt.conf
    - disregard all_proxy environment variable, like other methods

apt (0.9.7.9~exp2) experimental; urgency=low

[ Programs translations ]
  * Update all PO files and apt-all.pot
  * French translation completed (Christian Perrier)

[ Daniel Hartwig ]
  * cmdline/apt-get.cc:
    - do not have space between "-a" and option when cross building
      (closes: #703792)
  * test/integration/test-apt-get-download:
    - fix test now that #1098752 is fixed
  * po/{ca,cs,ru}.po:
    - fix merge artifact

[ David Kalnischkies ]
  * apt-pkg/indexcopy.cc:
    - rename RunGPGV to ExecGPGV and move it to apt-pkg/contrib/gpgv.cc
  * apt-pkg/contrib/gpgv.cc:
    - ExecGPGV is a method which should never return, so mark it as such
      and fix the inconsistency of returning in error cases
    - don't close stdout/stderr if it is also the statusfd
    - if ExecGPGV deals with a clear-signed file it will split this file
      into data and signatures, pass it to gpgv for verification
    - add method to open (maybe) clearsigned files transparently
  * apt-pkg/acquire-item.cc:
    - keep the last good InRelease file around just as we do it with
      Release.gpg in case the new one we download isn't good for us
  * apt-pkg/deb/debmetaindex.cc:
    - reenable InRelease by default
  * ftparchive/writer.cc,
    apt-pkg/deb/debindexfile.cc,
    apt-pkg/deb/deblistparser.cc:
    - use OpenMaybeClearSignedFile to be free from detecting and
      skipping clearsigning metadata in dsc and Release files

[ Michael Vogt ]
  * add regression test for CVE-2013-1051
  * implement GPGSplit() based on the idea from Ansgar Burchardt
    (many thanks!)
  * methods/connect.cc:
    - use Errno() instead of strerror(), thanks to David Kalnischk
  * doc/apt.conf.5.xml:
    - document Acquire::ForceIPv{4,6}

apt (0.9.7.9~exp1) experimental; urgency=low

[ Niels Thykier ]
  * test/libapt/assert.h, test/libapt/run-tests:
    - exit with status 1 on test failure

[ Daniel Hartwig ]
  * test/integration/framework:
    - continue after test failure but preserve exit status

[ Programs translation updates ]
  * Turkish (Mert Dirik). Closes: #703526

[ Colin Watson ]
  * methods/connect.cc:
    - provide useful error message in case of EAI_SYSTEM
      (closes: #703603)

[ Michael Vogt ]
  * add new config options "Acquire::ForceIPv4" and
    "Acquire::ForceIPv6" to allow focing one or the other
    (closes: #611891)
  * lp:~mvo/apt/fix-tagfile-hash:
    - fix false positives in pkgTagSection.Exists(), thanks to
      Niels Thykier for the testcase (closes: #703240)
    - this will require rebuilds of the clients as this used to
      be a inline function

apt (0.9.7.8) unstable; urgency=criticial

* SECURITY UPDATE: InRelease verification bypass
    - CVE-2013-1051

[ David Kalnischk ]
  * apt-pkg/deb/debmetaindex.cc,
    test/integration/test-bug-595691-empty-and-broken-archive-files,
    test/integration/test-releasefile-verification:
    - disable InRelease downloading until the verification issue is
      fixed, thanks to Ansgar Burchardt for finding the flaw

apt (0.9.7.8~exp2) experimental; urgency=low

* include two missing patches to really fix bug #696225, thanks to
    Guillem Jover
  * ensure sha512 is really used when available, thanks to Tyler Hicks
   (LP: #1098752)

apt (0.9.7.8~exp1) experimental; urgency=low

[ Manpages translation updates ]
  * Italian (Beatrice Torracca). Closes: #696601

[ Programs translation updates ]
  * Japanese (Kenshi Muto). Closes: #699783

[ Michael Vogt ]
  * fix pkgProblemResolver::Scores, thanks to Paul Wise.
    Closes: #697577
  * fix missing translated apt.8 manpages, thanks to Helge Kreutzmann
    for the report. Closes: #696923
  * apt-pkg/contrib/progress.cc:
    - Make "..." translatable to fix inconsistencies in the output
      of e.g. apt-get update. While this adds new translatable strings,
      not having translations for them will not break anything.
      Thanks to Guillem Jover. Closes: #696225
  * debian/apt.cron.daily:
    - when reading from /dev/urandom, use less entropy and fix a rare
      bug when the random number chksum is less than 1000.
      Closes: #695285
  * methods/https.cc:
    - reuse connection in https, thanks to Thomas Bushnell, BSG for the
      patch. LP: #1087543, Closes: #695359
    - add missing curl_easy_cleanup()
  * methods/http.cc:
    - quote spaces in filenames to ensure as the http method is also
      (potentially) used for non deb,dsc content that may contain
      spaces, thanks to Daniel Hartwig and Thomas Bushnell
      (LP: #1086997)
    - quote plus in filenames to work around a bug in the S3 server
      (LP: #1003633)
  * apt-pkg/indexrecords.cc:
    - support '\r' in the Release file

[ David Kalnischkies ]
  * apt-pkg/depcache.cc:
    - prefer to install packages which have an already installed M-A:same
      sibling while choosing providers (LP: #1130419)
 -- Michael Vogt <michael.vogt@ubuntu.com>   Wed, 10 Jul 2013 17:03:52 +0200

Changed in apt (Ubuntu):
status:	Fix Committed → Fix Released

Revision history for this message

Quanah Gibson-Mount (mishikal) wrote on 2015-11-03:

#10

I would note this is broken for https based repositories still.

Revision history for this message

Quanah Gibson-Mount (mishikal) wrote on 2015-11-03:

#11

I filed https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1512845 for the https issue

Ubuntu
apt package

Escape plus character in apt HTTP requests to work around Amazon S3 bug

Bug Description

Related branches

CVE References

Other bug subscribers

Patches

Remote bug watches

Ubuntuapt package

Escape plus character in apt HTTP requests to work around Amazon S3 bug

Bug Description

Related branches

CVE References

Other bug subscribers

Patches

Remote bug watches

Ubuntu
apt package