Scrapy Run From Python

I am trying to run Scrapy from Python. I'm looking at this code which (source): from twisted.internet import reactor from scrapy.crawler import Crawler from scrapy.settings import

Solution 1:

Just import it and pass to crawler.crawl(), like:

from testspiders.spiders.spider_a import MySpider

spider = MySpider()
crawler.crawl(spider)

Solution 2:

In Scrapy 0.19.x (may work with older versions) you can do the following.

spider = FollowAllSpider(domain='scrapinghub.com')
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run() # the script will block here

You can even call the command directly from a script like:

from scrapy import cmdline
cmdline.execute("scrapy crawl followall".split())  #followall is the spider's name

Take a look on my answer here. I changed the official documentation so now your crawler use your settings and can produce the output.