你好，我怎样才能让我的 crawlspider 工作，我可以登录，但没有任何反应，我真的没有抓到。我也一直在阅读 scrapy 文档，我真的不明白用于抓取的规则。为什么“登录成功，开始爬虫!”

我在 else 语句的末尾也有这条规则，但将其删除，因为它甚至没有被调用，因为它在我的 else block 中。所以我将它移到 start_request() 方法的顶部但出现错误所以我删除了我的规则。

 rules = (
                 Rule(extractor,callback='parse_item',follow=True),
                 )

我的代码:

from scrapy.contrib.spiders.init import InitSpider
from scrapy.http import Request, FormRequest
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.contrib.spiders import Rule
from scrapy.contrib.spiders import CrawlSpider, Rule

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector

from linkedconv.items import LinkedconvItem

class LinkedPySpider(CrawlSpider):
    name = 'LinkedPy'
    allowed_domains = ['linkedin.com']
    login_page = 'https://www.linkedin.com/uas/login'
   # start_urls = ["http://www.linkedin.com/csearch/results?type=companies&keywords=&pplSearchOrigin=GLHD&pageKey=member-home&search=Search#facets=pplSearchOrigin%3DFCTD%26keywords%3D%26search%3DSubmit%26facet_CS%3DC%26facet_I%3D80%26openFacets%3DJO%252CN%252CCS%252CNFR%252CF%252CCCR%252CI"]
    start_urls = ["http://www.linkedin.com/csearch/results"]


    def start_requests(self):
    yield Request(
    url=self.login_page,
    callback=self.login,
    dont_filter=True
    )

  #  def init_request(self):
    #"""This function is called before crawling starts."""
  #      return Request(url=self.login_page, callback=self.login)

    def login(self, response):
    #"""Generate a login request."""
    return FormRequest.from_response(response,
            formdata={'session_key': 'myemail@gmail.com', 'session_password': 'mypassword'},
            callback=self.check_login_response)

    def check_login_response(self, response):
    #"""Check the response returned by a login request to see if we aresuccessfully logged in."""
    if "Sign Out" in response.body:
        self.log("\n\n\nSuccessfully logged in. Let's start crawling!\n\n\n")
        # Now the crawling can begin..
        self.log('Hi, this is an item page! %s' % response.url)

        return 

    else:
        self.log("\n\n\nFailed, Bad times :(\n\n\n")
        # Something went wrong, we couldn't log in, so nothing happens.


    def parse_item(self, response):
    self.log("\n\n\n We got data! \n\n\n")
    self.log('Hi, this is an item page! %s' % response.url)
    hxs = HtmlXPathSelector(response)
    sites = hxs.select('//ol[@id=\'result-set\']/li')
    items = []
    for site in sites:
        item = LinkedconvItem()
        item['title'] = site.select('h2/a/text()').extract()
        item['link'] = site.select('h2/a/@href').extract()
        items.append(item)
    return items    

我的输出

C:\Users\ye831c\Documents\Big Data\Scrapy\linkedconv>scrapy crawl LinkedPy
2013-07-12 13:39:40-0500 [scrapy] INFO: Scrapy 0.16.5 started (bot: linkedconv)
2013-07-12 13:39:40-0500 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetCon
sole, CloseSpider, WebService, CoreStats, SpiderState
2013-07-12 13:39:41-0500 [scrapy] DEBUG: Enabled downloader middlewares: HttpAut
hMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, De
faultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMi
ddleware, ChunkedTransferMiddleware, DownloaderStats
2013-07-12 13:39:41-0500 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMi
ddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddle
ware
2013-07-12 13:39:41-0500 [scrapy] DEBUG: Enabled item pipelines:
2013-07-12 13:39:41-0500 [LinkedPy] INFO: Spider opened
2013-07-12 13:39:41-0500 [LinkedPy] INFO: Crawled 0 pages (at 0 pages/min), scra
ped 0 items (at 0 items/min)
2013-07-12 13:39:41-0500 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:602
3
2013-07-12 13:39:41-0500 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-07-12 13:39:41-0500 [LinkedPy] DEBUG: Crawled (200) <GET https://www.linked
in.com/uas/login> (referer: None)
2013-07-12 13:39:42-0500 [LinkedPy] DEBUG: Redirecting (302) to <GET http://www.
linkedin.com/nhome/> from <POST https://www.linkedin.com/uas/login-submit>
2013-07-12 13:39:45-0500 [LinkedPy] DEBUG: Crawled (200) <GET http://www.linkedi
n.com/nhome/> (referer: https://www.linkedin.com/uas/login)
2013-07-12 13:39:45-0500 [LinkedPy] DEBUG:


    Successfully logged in. Let's start crawling!



2013-07-12 13:39:45-0500 [LinkedPy] DEBUG: Hi, this is an item page! http://www.
linkedin.com/nhome/
2013-07-12 13:39:45-0500 [LinkedPy] INFO: Closing spider (finished)
2013-07-12 13:39:45-0500 [LinkedPy] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 1670,
     'downloader/request_count': 3,
     'downloader/request_method_count/GET': 2,
     'downloader/request_method_count/POST': 1,
     'downloader/response_bytes': 65218,
     'downloader/response_count': 3,
     'downloader/response_status_count/200': 2,
     'downloader/response_status_count/302': 1,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2013, 7, 12, 18, 39, 45, 136000),
     'log_count/DEBUG': 11,
     'log_count/INFO': 4,
     'request_depth_max': 1,
     'response_received_count': 2,
     'scheduler/dequeued': 3,
     'scheduler/dequeued/memory': 3,
     'scheduler/enqueued': 3,
     'scheduler/enqueued/memory': 3,
     'start_time': datetime.datetime(2013, 7, 12, 18, 39, 41, 50000)}
2013-07-12 13:39:45-0500 [LinkedPy] INFO: Spider closed (finished)

最佳答案

现在，爬行在 check_login_response() 结束，因为 Scrapy 没有被告知做更多的事情。

• 使用 start_requests() 对登录页面的第一个请求:OK
• 第二次请求发布登录信息:OK
• 使用 check_login_response 解析哪个响应...仅此而已

确实 check_login_response() 什么都不返回。为了让爬行继续进行，您需要返回 Request 实例(它告诉 Scrapy 接下来要获取哪些页面，请参阅 Scrapy 关于蜘蛛回调的文档)

因此，在 check_login_response() 中，您需要返回一个 Request 实例到包含您接下来要抓取的链接的起始页，可能是您要抓取的一些 URL在 start_urls 中定义。

    def check_login_response(self, response):
        #"""Check the response returned by a login request to see if we aresuccessfully logged in."""
        if "Sign Out" in response.body:
            self.log("\n\n\nSuccessfully logged in. Let's start crawling!\n\n\n")
            # Now the crawling can begin..
            return Request(url='http://linkedin.com/page/containing/links')

默认情况下，如果您没有为您的Request 设置回调，蜘蛛会调用它的parse() 方法(http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.spider.BaseSpider.parse)。

在您的情况下，它将自动为您调用CrawlSpider 的内置parse() 方法，该方法应用Rule您已定义获取下一页。

您必须在蜘蛛类的 rules 属性中定义您的 CrawlSpider 规则，就像您为 name、allowed_domain 所做的那样 等，处于同一级别。

http://doc.scrapy.org/en/latest/topics/spiders.html#crawlspider-example提供示例规则。主要思想是使用 allow 中的正则表达式告诉提取器您对页面中的哪种绝对 URL 感兴趣。如果您未在 SgmlLinkExtractor 中设置 allow，它将匹配所有链接。

并且每个规则都应该有一个用于这些链接的回调，在您的例子中是 parse_item()。

祝你解析 LinkedIn 页面好运，我认为页面中的很多内容都是通过 Javascript 生成的，可能不在 Scrapy 获取的 HTML 内容中。

关于html - 使用 scrapy 获取 crawlspider 来处理经过身份验证(登录)的用户 session ，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/17622097/