From Scrapy.selector Import Selector Error

I am unable to do the following: from scrapy.selector import Selector The error is: File '/Desktop/KSL/KSL/spiders/spider.py', line 1, in from scrapy.selector import Selector

Solution 1:

Try importing HtmlXPathSelector instead.

    from scrapy.selector import HtmlXPathSelector

And then use the .select() method to parse out your html. For example,

    sel = HtmlXPathSelector(response)
    site_names = sel.select('//ul/li')

If you are following the tutorial on the Scrapy site (http://doc.scrapy.org/en/latest/intro/tutorial.html), the updated example would look like this:

    from scrapy.spider import BaseSpider
    from scrapy.selector import HtmlXPathSelector

    class DmozSpider(BaseSpider):
        name = "dmoz"
        allowed_domains = ["dmoz.org"]
        start_urls = [
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
        ]

        def parse(self, response):
            sel = HtmlXPathSelector(response)
            sites = sel.select('//ul/li')

            for site in sites:
                title = site.select('a/text()').extract()
                link = site.select('a/@href').extract()
                desc = site.select('text()').extract()
                print title, link, desc

Hope this helps!

Solution 2:

I encounter the same problem. I think there is something wrong with your scrapy version.

You could type scrapy version -v into cmd to check the version. As far as I know, the newest version is 0.24.4 (2014.10.23). You could visit http://scrapy.org/ to find the newest.

Solution 3:

Had the same issue which was due to the way I installed scrapy (my OS is Ubuntu). I did it through

sudo apt-get install python-scrapy

Install it from python instead. If you use Anaconda just do

conda install -c scrapinghub scrapy

If not, then

pip install Scrapy

http://doc.scrapy.org/en/latest/intro/install.html