Recommended packages are excessive

The Scrapy version in the Ubuntu/Debian python-scrapy package is quite old. I suggest you use the Ubuntu packages provided by the Scrapy project instead: http://doc.scrapy.org/topics/ubuntu.html (APT repo available too).

It would be great if we could synchronize the Debian package provided by the Scrapy project with the one available in Ubuntu/Debian, but not I'm not sure what's the procedure for that.

Hello,

python-guppy is recommended by python-scrapy because one of it files tries to import it [1]:
/
| try:
| import guppy
| hpy = guppy.hpy()
| except ImportError:
| hpy = None
\

As for the not up-to-date package in Debian/Ubuntu, it is due to the Debian Freeze. I have uploaded the 0.9 version of python-scrapy to experimental, as it does not fit to be in squeeze (due to the freeze). I will work on packaging the latest version and upload it on experimental (Debian). Then, a synchronization request can be sent on launchpad asking the Ubuntu Teams to review, and upload the package onto Ubuntu repositories.

If you need any help on this part, I would be glad to give you a hand.

Cheers,
 Ignace M

[1] http://hg.scrapy.org/scrapy/file/8ffd63e657f0/scrapy/telnet.py
[2] http://packages.qa.debian.org/p/python-scrapy/news/20100823T164725Z.html

I am closing this bug setting it as "Invalid". Please reopen it if you it necessary.

Regards,
 Ignace M

I see that guppy is imported there...but what about requiring python-django? Surely, a scraping utility doesn't need django installed, right?

I simple grep shows that it does:
----
$ grep -R django scrapy/
scrapy/tests/test_djangoitem/models.py:from django.db import models
scrapy/tests/test_djangoitem/models.py: app_label = 'test_djangoitem'
scrapy/tests/test_djangoitem/__init__.py:from scrapy.contrib_exp.djangoitem import DjangoItem, Field
scrapy/tests/test_djangoitem/__init__.py:os.environ['DJANGO_SETTINGS_MODULE'] = 'scrapy.tests.test_djangoitem.settings'
scrapy/tests/test_djangoitem/__init__.py: import django
scrapy/tests/test_djangoitem/__init__.py: django = None
scrapy/tests/test_djangoitem/__init__.py:if django:
scrapy/tests/test_djangoitem/__init__.py: django_model = Person
scrapy/tests/test_djangoitem/__init__.py: if not django:
scrapy/contrib_exp/djangoitem.py: if cls.django_model:
scrapy/contrib_exp/djangoitem.py: cls._model_meta = cls.django_model._meta
scrapy/contrib_exp/djangoitem.py: django_model = None
scrapy/contrib_exp/djangoitem.py: model = self.django_model(**modelargs)
----

However, it might be interesting to 'suggest' it instead 'recommend' it. I'll work on this in the next release.

Cheers,
 Ignace M

Scrapy contains many additional code in scrapy.contrib which are extras and not required to run the framework. boto is another example of such dependency, which is even listed as depends (not recommends). The only 3 depends are: python-libxml2, python-twisted, python-openssl (this is optional but highly recommended and it's listed as suggests now).

I would list as recommended: python-pygments, python-imaging (used by a very popular contrib - the images pipeline), and maybe python-guppy (or perhaps in suggests).

Packages I can't decide whether to put in suggests or recommends:

* ipython (supports an improved shell console - we should probably do the same as django?)
* python-boto (enables S3 backend for images pipeline)
* python-pygments (supports colorized output in scrapy parse command)
* python-guppy (enhances memory debugging capabilities from the telnet console)

python-django doesn't fit in either recommends or suggests, IMO.

Thanks Pablo for your input!

I will take all of your remarks into account in the upcoming package.

Regards,
 Ignace M

iPython shouldn't be included. It's a tool that's useful for Python developers. Surely, if you installed a scraping utility, you wouldn't expect typing the word python into a console to change, right? I'd be dumbfounded if that happened to me and I wasn't watching carefully what I was installing.

Principle of least surprise.

mlissner, bear in mind that installing ipython doesn't replace the "python" command, but adds an "ipython" command instead. However, applications can optionally choose to embed ipython shells (when ipython is available) and otherwise fallback to regular python shell. This is what Django and Scrapy do.

It could still affect other applications that follow this behaviour, but it's not as critical as replacing a console command.

Ignace, thanks for taking care of this. We have just released Scrapy 0.14, so you may want to bump the version as part of the changes for the upcoming package. Bear in mind that Scrapy now depends on a separate package: w3lib, so you would have to add a python-w3lib package as well.

Perhaps this is a good time to synchronize Ubuntu official package with Scrapy official package [1]. But it's probably worth creating a separate ticket to discuss that.

[1] https://github.com/scrapy/scrapy/tree/master/debian

Pablo, true, but at least in Ubuntu, when you install iPython, it replaces the python command with itself, I think. Maybe that's just the django shell...not positive.

On Thu, Nov 17, 2011 at 15:21, Pablo Hoffman <email address hidden> wrote:
> Ignace, thanks for taking care of this. We have just released Scrapy
> 0.14, so you may want to bump the version as part of the changes for the
> upcoming package. Bear in mind that Scrapy now depends on a separate
> package: w3lib, so you would have to add a python-w3lib package as well.
>
> Perhaps this is a good time to synchronize Ubuntu official package with
> Scrapy official package [1].  But it's probably worth creating a
> separate ticket to discuss that.
>
> [1] https://github.com/scrapy/scrapy/tree/master/debian

Sure! I will work on it by the end of next week, I'll keep you posted.

Cheers,
 Ignace M