pycurl reduces smart download speed significantly

The remainder of the report:

msg1166 (view) Author: niemeyer Date: 2007-07-05.01:26:21

Oh my.. please don't paste that in a public forum bogdano. :-) That's
a programming atrocity I've made while looking to implement support
for protocols quickly.

I hope to fix the way that the fetcher behaves internally in the
near future.

msg1157 (view) Author: bogdano Date: 2007-06-25.00:57:21

On Mon, Jun 25, 2007 at 12:26:43AM +0000, Tobias Gerschner at Labix Tracker wrote:
>
> Tobias Gerschner <email address hidden> added the comment:
>
> This has been reported from several machines and is not related to proxy use at
> all. We added pycurl by default so that people don't have issues using proxies.
> Anyone having pycurl installed experienced noticeably slower download speed. I
> won't exclude packaging error on our side , but I have found none so far and
> think it was time to get this discussion upstream started.
>

Noticed the same behavior here some time ago. I didn't investigated how to
properly fix it yet, but the following patch increased the transfer rate:

--- smart/fetcher.py (revision 876)
+++ smart/fetcher.py (working copy)
@@ -1668,7 +1668,7 @@
             while res == mp:
                 res, num = multi.perform()
             self._lock.release()
- time.sleep(0.2)
+ #time.sleep(0.2)
         self._running = False

Seems to not be the right way to fix it. Needs further investigation.

msg1156 (view) Author: tobig Date: 2007-06-25.00:26:42

Sorry for the incomplete report. The issue is quite easy to reproduce here:

1) Package download from a near mirror happens at 500 kbyte/s using smart
without having pycurl installed.
2) uninstall the package and install pycurl
3) install the same package again and the download speed is always very low ( 60
kbyte/s ).

This has been reported from several machines and is not related to proxy use at
all. We added pycurl by default so that people don't have issues using proxies.
Anyone having pycurl installed experienced noticeably slower download speed. I
won't exclude packaging error on our side , but I have found none so far and
think it was time to get this discussion upstream started.

msg1151 (view) Author: peter-endian Date: 2007-06-21.13:27:05

i noticed exactly the contrary. pycurl is way faster than urllib, which of
course is not very surprising
Do you face this only with the proxy within the path? Then it's quite normal. A
proxy always slows down the connection.

I tested this on Fedora 8 (actually not this exactly but I installed pycurl). I noticed the following: without pycurl I get 300-400k/s and with pycurl I get about 80k/s for each server in the mirrorlist. So with five packages downloading I get about 70-80k/s for each package as long as they are on different servers and this reduces if multiple packages are coming from the same server. So if only one package is being downloaded then it gets about 80k/s.

Rehan,

I did a little bit more testing and found that my download rates with pycurl are inversely proportional to the length of the sleep that you identified. Why is removing (or shortening) that sleep correct?

Thanks,

Michael

Rather, why is it /not/ correct?

Switched from time.sleep to multi.select, so will only sleep if there is no data.