package-data-downloader utility does not honor apt http proxy settings

Status changed to 'Confirmed' because the bug affects multiple users.

The reason this is occurring in precise is that the flashplugin-installer was converted to use the update-notifier package-data-downloader hook which unfortunately does not take the apt proxy configuration settings into account.

In versions before precise, the script that did this formerly in the postinst did the following:

   APT_PROXIES=$(apt-config shell \
                http_proxy Acquire::http::Proxy \
                https_proxy Acquire::https::Proxy \
                ftp_proxy Acquire::ftp::Proxy \
   )

  if [ -n "$APT_PROXIES" ]; then
                eval export $APT_PROXIES
  fi

The package-data-downloader utility should do something similar. Moving this bug over to the update-notifier package.

Sorry, but this is by design. See bug #979477 for the reasoning.

We need to have consistent behavior for how we handle proxies for data downloads; pointing at the apt proxy for downloading things that are not .deb packages (and not packages that are part of the redistributable archive) is going to cause failures in some configurations. The only generally correct solution here seems to be to use the system's proxy settings, instead of the apt proxy settings.

So you can specify a system proxy by setting the http_proxy variable in /etc/environment. Depending on how you're gaining root privileges for apt, you may still run into problems due to bug #982684, which will hopefully be fixed soon.

I've been trying to debug this for hours. I just upgraded to Precise from Oneiric. My upgrade hung when attempting to configure ttf-mscorefonts-installer. I seem to have completed the upgrade successfully by aborting, removing ttf-mscorefonts-installer and flashplugin-installer and running dpkg --configure -a. I have rebooted, and everything seems fine.

Now I can't reinstall flashplugin-installer. It gets to the external file download, and hangs. It hangs in apt-get, it hangs in dpkg and it hangs in Synaptic. You say that the new download method is supposed to pick up the proxy variable from /etc/environment, but I *have* the proxy variable set in /etc/environment (and in every other possible place that I could think of) and it is not being detected.

I have added 'Defaults env_keep = "http_proxy ftp_proxy https_proxy no_proxy"' to my sudoers file (above env_reset), so the http_proxy variable should be kept. in any case, I get the exact same problem if I try to install the package as root instead of using sudo (after verifying that http_proxy is set in root's environment). So this doesn't sound like bug #982684, unless I'm misunderstanding exactly what that bug affects.

I can download the file without any problems manually. As a workaround, would it be possible for me to put this file in the correct download directory, and have the script detect it as already downloaded? At the moment it appears to be literally impossible for me to install these packages. I cannot access the outside world without the proxy.

@Adrianna: Try the following work-arounds:

To download flashplugin for the first time (or after removing the package):
$ sudo -i apt-get install flashplugin-installer

If you have pending downloads, either from an update or a previous failed attempt, this can be fixed with:
$ sudo -i /usr/lib/update-notifier/package-data-downloader

No luck. Both commands hang at the download. However, I have discovered that the problem is not with finding the proxy settings (which are found correctly -- it may not even be necessary to explicitly set the proxies from the environment; this seems to be urllib's default behaviour). The script does download the file to a location in /tmp/ -- but it doesn't do anything once the download is complete.

I made a minimal test case in python which reproduces this behaviour. My script completes the download and exits normally if I change the url to the Google homepage, but fails to exit if the url is the location of the flash tarball. I don't know whether this is related to the file size or the server -- if I try to download a much smaller file from the same directory in the Canonical archive it doesn't work. If I upload the tarball to another rempte server and try to download it from there, it still doesn't work.

I can use my test case to download the tarball from my local server, but only if I also unset all my proxy environment variables (and don't set them in urllib). (I don't know why that is necessary; I use the local network's domain name for my local server, and I can download the file with wget with all the environment variables set).

I suppose that I could use this as an elaborate workaround to install both the necessary packages, but I would really like to understand the underlying cause of the problem.

My proxy is cntlm. It shouldn't require authentication (it takes care of the authentication required by the upstream NTLM proxy).

Hi,

I am using an HTTP proxy and am also affected:

$ sudo -i /usr/lib/update-notifier/package-data-downloader
flashplugin-installer: downloading http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_11.2.202.235.orig.tar.gz
Traceback (most recent call last):
  File "/usr/lib/update-notifier/package-data-downloader", line 234, in process_download_requests
    dest_file = urllib.urlretrieve(files[i])[0]
  File "/usr/lib/python2.7/urllib.py", line 93, in urlretrieve
    return _urlopener.retrieve(url, filename, reporthook, data)
  File "/usr/lib/python2.7/urllib.py", line 239, in retrieve
    fp = self.open(url, data)
  File "/usr/lib/python2.7/urllib.py", line 207, in open
    return getattr(self, name)(url)
  File "/usr/lib/python2.7/urllib.py", line 344, in open_http
    h.endheaders(data)
  File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 814, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 776, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 757, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
    raise err
IOError: [Errno socket error] [Errno 110] Connection timed out

$ grep proxy /etc/environment
https_proxy = http://webcache.name.fr:3128/
http_proxy = http://webcache.name.fr:3128/
ftp_proxy = http://webcache.name.fr:3128/
no_proxy='localhost,127.0.0.0/8,*.groupe.name.fr'

$ sudo -i set|grep proxy
no_proxy='localhost,127.0.0.0/8,*.groupe.name.fr'

@pdecat: Something wrong there.

$ sudo -i set|grep proxy

...shows my proxies correctly. Did you click 'Apply system wide' when setting the proxies in the network settings applet?

For the record, I don't get a "connection timed out" or any other kind of error message -- if I interrupt the downloader with ctrl-c I get this traceback:

$ sudo -i /usr/lib/update-notifier/package-data-downloader
flashplugin-installer: downloading http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_11.2.202.235.orig.tar.gz
^CTraceback (most recent call last):
  File "/usr/lib/update-notifier/package-data-downloader", line 296, in <module>
    process_download_requests()
  File "/usr/lib/update-notifier/package-data-downloader", line 234, in process_download_requests
    dest_file = urllib.urlretrieve(files[i])[0]
  File "/usr/lib/python2.7/urllib.py", line 93, in urlretrieve
    return _urlopener.retrieve(url, filename, reporthook, data)
  File "/usr/lib/python2.7/urllib.py", line 267, in retrieve
    block = fp.read(bs)
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
KeyboardInterrupt

Sometimes the traceback only goes as far as 'block = fp.read(bs)'.

Before I interrupt the downloader, I can observe the temporary downloaded file growing in /tmp/ until it reaches the full expected size. Then nothing happens. The downloader doesn't appear to detect that the download was successful. I'm assuming that this is some kind of problem in urllib.

I have solved my problem by editing the download script to use wget in a subprocess instead of urllib.urlretrieve. It's a bit of a hack, and maybe it's not a viable long-term solution, but now everything works perfectly for me. I was able to install both flash and the MS core fonts. I'm attaching a diff showing what I did.

@ginggs I just reapplied the settings system wide using the graphical user interface and it fixed it, thanks.

The reason I edited /etc/environment by hand and did not use the GUI in the first place is that it does not allow to declare exceptions (the 'no_proxy' variable).

Here are the logs (Note the now duplicated entries in /etc/environment) :

$ cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games"
https_proxy = http://webcache.name.fr:3128/
http_proxy = http://webcache.name.fr:3128/
ftp_proxy = http://webcache.name.fr:3128/
no_proxy='localhost,127.0.0.0/8,*.groupe.name.fr'

http_proxy="http://webcache.name.fr:3128/"
https_proxy="https://webcache.name.fr:3128/"
ftp_proxy="ftp://webcache.name.fr:3128/"

$ sudo -i set|grep proxy
ftp_proxy=ftp://webcache.name.fr:3128/
http_proxy=http://webcache.name.fr:3128/
https_proxy=https://webcache.name.fr:3128/
no_proxy='localhost,127.0.0.0/8,*.groupe.name.fr'

$ sudo -i /usr/lib/update-notifier/package-data-downloader
flashplugin-installer: downloading http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_11.2.202.235.orig.tar.gz
Installing from local file /tmp/tmpVDNyU4.gz
Flash Plugin installed.

@Adrianna: I'm confirming that when using cntlm, package-data-downloader downloads the file but never installs it, no error messages either.
When using our corporate proxy, package-data-downloader downloads the file and installs it successfully.

I arranged with our proxy administrators for an authentication-bypass rule so that we can access http://archive.canonical.com/ through the proxy without requiring authentication.

@pdecat: Glad that fixed it for you. Yes, the 'gnome-control-center network' applet is lacking in functionality, see #861443.

@pdecat: Actually, the following are better examples:

gnome-control-center doesn't allow to specify proxy exceptions
http://bugs.launchpad.net/ubuntu/+source/gnome-control-center/+bug/843268

No Profiles for Network Proxy
http://bugs.launchpad.net/ubuntu/+source/gnome-control-center/+bug/844535

Is there any news on a fix for this? I understand the desire for consistency in how proxy settings are used, but breaking things isn't good either. 12.04 is an LTS -- and I'm very surprised this isn't getting attention to be fixed. Flash is way too common to ignore. For those of us who have to use a proxy, this is rather aggravating.

I am affected by something looking like this bug even though I am on 12.10. The strange thing is that the flashplugin-installer second stage download works when updating from:
- dpkg command line as root
- dpkg command line as sudo from user account
- synaptics apply
but not from:
- update-notifier

From inside the update-notifier GUI I get the following log:
Processing triggers for update-notifier-common ...
flashplugin-installer: downloading http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_11.2.202.270.orig.tar.gz
Traceback (most recent call last):
  File "/usr/lib/update-notifier/package-data-downloader", line 234, in process_download_requests
    dest_file = urllib.urlretrieve(files[i])[0]
  File "/usr/lib/python2.7/urllib.py", line 93, in urlretrieve
    return _urlopener.retrieve(url, filename, reporthook, data)
  File "/usr/lib/python2.7/urllib.py", line 239, in retrieve
    fp = self.open(url, data)
  File "/usr/lib/python2.7/urllib.py", line 207, in open
    return getattr(self, name)(url)
  File "/usr/lib/python2.7/urllib.py", line 344, in open_http
    h.endheaders(data)
  File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 818, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 780, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 761, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
    raise err
IOError: [Errno socket error] [Errno 110] Connection timed out

The failing process has the following parent chain:
  /sbin/init
    /usr/bin/python3.2 /usr/sbin/aptd
      /usr/bin/python3.2 /usr/sbin/aptd
        /usr/bin/dpkg --status-fd 75 --unpack --auto-deconfigure /var/cache/apt/archives/flashplugininstaller_11.2.202.270ubuntu0.12.10.1_amd64.deb
          /bin/sh /var/lib/dpkg/info/update-notifier-common.postinst triggered /usr/share/package-data-downloads
            /usr/bin/python /usr/lib/update-notifier/package-data-downloader
and I have inspected environment variables in the last two processes, and there are no *_proxy variables in them when triggered by update-notifier.
When triggered in the other ways the variables are present.

My /etc/environment:
all_proxy=...
http_proxy=...

My /etc/sudoers.d/allow-proxy-vars-in-sudo:
Defaults env_keep += all_proxy
Defaults env_keep += http_proxy

sudo env | grep proxy:
http_proxy=...
all_proxy=...

Can someone shed some light on why this is happening?

I can confirm the description of mikewse in comment #16.

If I get an update of flashplugin-installer via update-notifier, the update fails because the proxy settings are not used. I then ran:

sudo dpkg --purge flashplugin-installer; sudo apt-get install flashplugin-installer

and the package download works just fine.

My /etc/environment has settings for (example.com inserted in place of real domain):

http_proxy="http://proxy.example.com:3128/"
ftp_proxy="ftp://proxy.example.com:3128/"
https_proxy="https://proxy.example.com:3128/"
socks_proxy="socks://proxy.example.com:3128/"

In addition, the same proxies are configured in /etc/apt/apt.conf:

Acquire::http::proxy "http://proxy.example.com:3128/";
Acquire::ftp::proxy "ftp://proxy.example.com:3128/";
Acquire::https::proxy "https://proxy.example.com:3128/";
Acquire::socks::proxy "socks://proxy.example.com:3128/";

These environment variables are available in both "sudo -s" and "sudo -i".

The proxy was configured via the network settings tool as manual configuration and applied system-wide.

It seems that update-notifier does not provide these proxy settings to package-data-downloader.

Just to show a workaround. It's really *UGLY* but worked for me:

1.- Download manually
http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_xx.x.x.x.orig.tar.gz

2.- Make you root (sudo -i, su -, etc)

3.- Make a backup of /usr/bin/update-notifier/package-data-downloader:
cp /usr/bin/update-notifier/package-data-downloader /usr/bin/update-notifier/package-data-downloader.bak

4.- Edit /usr/bin/update-notifier/package-data-downloader and replace this line:

dest_file = urllib.urlretrieve(files[i])[0]

by this one:

dest_file = urllib.urlretrieve("/path/to/your/file/adobe-flashplugin_xx.x.x.x.orig.tar.gz")[0]

5.- Reinstall flashplugin-installer package:
apt-get install --reinstall flashplugin-installer

6.- Recover the original package-data-downloader file:
mv /usr/bin/update-notifier/package-data-downloader.bak /usr/bin/update-notifier/package-data-downloader

This should work on the command line (not via CRON or GUI)

export http_proxy=http://proxy.mydomain.edu:3128/
apt-get --reinstall install flashplugin-installer

/usr/lib/update-notifier/package-data-downloader from update-notifier uses python urllib

It seems to be problems when using proxies. In addition, urllib doesn't supports https protocol.

https://docs.python.org/2/library/urllib.html#urllib-restrictions

Using a http proxy, adobe-flashplugin updates will work but not allways:

urllib.urlretrieve("http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_11.2.202.425.orig.tar.gz", "adobe-flashplugin.tar.gz")

Using a https proxy, pepflashpluguin updates will never work:

urllib.urlretrieve("https://dl-ssl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_39.0.2171.95-1_i386.deb", "google-chrome.deb")

So, scripts must be something like:

export http_proxy=http://192.168.0.1:3128
export https_proxy=https://192.168.0.1:3128
apt-get install -y flashplugin-installer
apt-get install -y pepflashplugin-installer
echo "" >> /etc/chromium-browser/default
echo ". /usr/lib/pepflashplugin-installer/pepflashplayer.sh" >> /etc/chromium-browser/default
echo "" >> /etc/chromium-browser/default
unset http_proxy
unset https_proxy
apt-get --reinstall install -y flashplugin-installer
apt-get --reinstall install -y pepflashplugin-installer

Of course, if yo do this, your corporative firewall/proxy must allow direct connections to archive.canonical.com (http) and dl-ssl.google.com (https).

In my opinion, update-notifier (and other applications using urllib) should avoid python urllib

There are many bugs related (in fact, they are duplicated):

https://bugs.launchpad.net/ubuntu/+source/anacron/+bug/1209146
https://bugs.launchpad.net/ubuntu/+source/flashplugin-nonfree/+bug/1098233
https://bugs.launchpad.net/ubuntu/+source/flashplugin-nonfree/+bug/1037662
https://bugs.launchpad.net/ubuntu/+source/update-notifier/+bug/1005837
https://bugs.launchpad.net/ubuntu/+source/update-notifier/+bug/983559
https://bugs.launchpad.net/zc.buildout/+bug/484735