您好，我在我的 Mac 上安装了 Python Scrapy，我正在尝试关注 v ery first example在他们的网站上。

他们试图运行命令:

scrapy crawl mininova.org -o scraped_data.json -t json

我不太明白这是什么意思？看起来scrapy原来是一个单独的程序。而且我认为他们没有名为 crawl 的命令。在示例中，他们有一段代码，即 MininovaSpider 类和 TorrentItem 的定义。我不知道这两个类应该去哪里，去同一个文件，这个python文件的名字是什么？

最佳答案

TL；DR:见Self-contained minimum example script to run scrapy .

首先，有一个普通的Scrapy项目，有一个单独的.cfg、settings.py、pipelines.py、 items.py、spiders 包等是保留和处理网络抓取逻辑的推荐方法。它提供了模块化、关注点分离，使事物保持有序、清晰和可测试。

如果您关注 official Scrapy tutorial要创建一个项目，您正在通过一个特殊的 scrapy 命令行工具运行网络抓取:

scrapy crawl myspider

但是，Scrapy 也提供了一个 API 到 run crawling from a script .

有几个关键概念需要提一下:

• Settings class - 基本上是一个用default built-in values 初始化的键值“容器”
• Crawler class - 主类，其作用类似于使用 Scrapy 进行网络抓取所涉及的所有不同组件的粘合剂
• Twisted reactor - 因为 Scrapy 内置在 twisted 异步网络库之上 - 要启动爬虫，我们需要将它放在 Twisted Reactor 中，简单来说就是一个事件循环:

The reactor is the core of the event loop within Twisted – the loop which drives applications using Twisted. The event loop is a programming construct that waits for and dispatches events or messages in a program. It works by calling some internal or external “event provider”, which generally blocks until an event has arrived, and then calls the relevant event handler (“dispatches the event”). The reactor provides basic interfaces to a number of services, including network communications, threading, and event dispatching.

这是从脚本运行 Scrapy 的一个基本且简化的过程:

• 创建一个 Settings 实例(或使用 get_project_settings() 来使用现有设置):

  settings = Settings()  # or settings = get_project_settings()
  

• 用传入的settings实例实例化Crawler:

  crawler = Crawler(settings)
  

• 实例化一个蜘蛛(这就是最终的意义，对吧？):

  spider = MySpider()
  

• 配置信号。如果你想要一个后处理逻辑，这是一个重要的步骤，collect stats或者，至少，要完成爬网，因为需要手动停止扭曲的 reactor。 Scrapy 文档建议停止 spider_closed signal 中的 reactor处理程序:

Note that you will also have to shutdown the Twisted reactor yourself after the spider is finished. This can be achieved by connecting a handler to the signals.spider_closed signal.

def callback(spider, reason):
    stats = spider.crawler.stats.get_stats()
    # stats here is a dictionary of crawling stats that you usually see on the console        

    # here we need to stop the reactor
    reactor.stop()

crawler.signals.connect(callback, signal=signals.spider_closed)

• 使用传入的蜘蛛配置并启动爬虫实例:

  crawler.configure()
  crawler.crawl(spider)
  crawler.start()
  

• 可选择开始logging :

  log.start()
  

• 启动 react 器 - 这会阻止脚本执行:

  reactor.run()
  

这是一个使用 DmozSpider spider 的独立脚本示例并涉及item loaders与 input and output processors和 item pipelines :

import json

from scrapy.crawler import Crawler
from scrapy.contrib.loader import ItemLoader
from scrapy.contrib.loader.processor import Join, MapCompose, TakeFirst
from scrapy import log, signals, Spider, Item, Field
from scrapy.settings import Settings
from twisted.internet import reactor


# define an item class
class DmozItem(Item):
    title = Field()
    link = Field()
    desc = Field()


# define an item loader with input and output processors
class DmozItemLoader(ItemLoader):
    default_input_processor = MapCompose(unicode.strip)
    default_output_processor = TakeFirst()

    desc_out = Join()


# define a pipeline
class JsonWriterPipeline(object):
    def __init__(self):
        self.file = open('items.jl', 'wb')

    def process_item(self, item, spider):
        line = json.dumps(dict(item)) + "\n"
        self.file.write(line)
        return item


# define a spider
class DmozSpider(Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        for sel in response.xpath('//ul/li'):
            loader = DmozItemLoader(DmozItem(), selector=sel, response=response)
            loader.add_xpath('title', 'a/text()')
            loader.add_xpath('link', 'a/@href')
            loader.add_xpath('desc', 'text()')
            yield loader.load_item()


# callback fired when the spider is closed
def callback(spider, reason):
    stats = spider.crawler.stats.get_stats()  # collect/log stats?

    # stop the reactor
    reactor.stop()


# instantiate settings and provide a custom configuration
settings = Settings()
settings.set('ITEM_PIPELINES', {
    '__main__.JsonWriterPipeline': 100
})

# instantiate a crawler passing in settings
crawler = Crawler(settings)

# instantiate a spider
spider = DmozSpider()

# configure signals
crawler.signals.connect(callback, signal=signals.spider_closed)

# configure and start the crawler
crawler.configure()
crawler.crawl(spider)
crawler.start()

# start logging
log.start()

# start the reactor (blocks execution)
reactor.run()

以通常的方式运行它:

python runner.py

并在管道的帮助下观察导出到 items.jl 的项目:

{"desc": "", "link": "/", "title": "Top"}
{"link": "/Computers/", "title": "Computers"}
{"link": "/Computers/Programming/", "title": "Programming"}
{"link": "/Computers/Programming/Languages/", "title": "Languages"}
{"link": "/Computers/Programming/Languages/Python/", "title": "Python"}
...

此处提供 Gist(请随时改进):

• Self-contained minimum example script to run scrapy

注意事项:

如果您通过实例化 Settings() 对象来定义 settings - 您将获得所有默认的 Scrapy 设置。但是，例如，如果您想配置现有管道，或配置 DEPTH_LIMIT 或调整任何其他设置，则需要通过 settings.set() 在脚本中进行设置。 (如示例所示):

pipelines = {
    'mypackage.pipelines.FilterPipeline': 100,
    'mypackage.pipelines.MySQLPipeline': 200
}
settings.set('ITEM_PIPELINES', pipelines, priority='cmdline')

或者，使用现有的 settings.py 并预先配置所有自定义设置:

from scrapy.utils.project import get_project_settings

settings = get_project_settings()

关于该主题的其他有用链接:

• How to run Scrapy from within a Python script
• Confused about running Scrapy from within a Python script
• scrapy run spider from script

关于python - Scrapy 非常基本的例子，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/18838494/