我正在学习 Python 并试图抓取这个 page用于下拉菜单中的特定值。之后，我需要单击结果表中的每个项目以检索特定信息。我能够选择该项目并检索网络驱动程序上的信息。但是我不知道如何将响应url传递给crawlspider。

driver = webdriver.Firefox()
driver.get('http://www.cppcc.gov.cn/CMS/icms/project1/cppcc/wylibary/wjWeiYuanList.jsp')
more_btn = WebDriverWait(driver, 20).until(
     EC.visibility_of_element_located((By.ID, '_button_select'))
            )  
more_btn.click()

## select specific value from the dropdown
driver.find_element_by_css_selector("select#tabJcwyxt_jiebie >     option[value='teyaoxgrs']").click()
driver.find_element_by_css_selector("select#tabJcwyxt_jieci > option[value='d11jie']").click()
search2 = driver.find_element_by_class_name('input_a2')
search2.click()
time.sleep(5)

## convert html to "nice format"
text_html=driver.page_source.encode('utf-8')
html_str=str(text_html)

## this is a hack that initiates a "TextResponse" object (taken from the Scrapy module)
resp_for_scrapy=TextResponse('none',200,{},html_str,[],None)

## convert html to "nice format"
text_html=driver.page_source.encode('utf-8')
html_str=str(text_html)

resp_for_scrapy=TextResponse('none',200,{},html_str,[],None)

所以这就是我被困的地方。我能够使用上面的代码进行查询。但是如何将 resp_for_scrapy 传递给 crawlspider？我用 resp_for_scrapy 代替了 item 但这没有用。

## spider 
class ProfileSpider(CrawlSpider):
name = 'pccprofile2'
allowed_domains = ['cppcc.gov.cn']
start_urls = ['http://www.cppcc.gov.cn/CMS/icms/project1/cppcc/wylibary/wjWeiYuanList.jsp']    

def parse(self, resp_for_scrapy):

    hxs = HtmlXPathSelector(resp_for_scrapy)
    for post in resp_for_scrapy.xpath('//div[@class="table"]//ul//li'):
        items = []
        item = Ppcprofile2Item()
        item ["name"] = hxs.select("//h1/text()").extract()
        item ["title"] = hxs.select("//div[@id='contentbody']//tr//td//text()").extract()
        items.append(item)

    ##click next page      
    while True:
        next = self.driver.findElement(By.linkText("下一页"))
        try:
            next.click()
        except:
            break

    return(items)

如有任何建议，我们将不胜感激!!!!

编辑 我在蜘蛛类之前包括了一个中间件类以从下拉列表中进行选择。但是现在没有错误，也没有结果。

class JSMiddleware(object):
    def process_request(self, request, spider):
        driver = webdriver.PhantomJS()
         driver.get('http://www.cppcc.gov.cn/CMS/icms/project1/cppcc/wylibary/wjWeiYuanList.jsp')


    # select from the dropdown
        more_btn = WebDriverWait(driver, 20).until(
        EC.visibility_of_element_located((By.ID, '_button_select'))
                )
        more_btn.click()


        driver.find_element_by_css_selector("select#tabJcwyxt_jiebie > option[value='teyaoxgrs']").click()
        driver.find_element_by_css_selector("select#tabJcwyxt_jieci > option[value='d11jie']").click()
        search2 = driver.find_element_by_class_name('input_a2')
        search2.click()
        time.sleep(5)

        #get the response 
        body = driver.page_source
        return HtmlResponse(driver.current_url, body=body, encoding='utf-8', request=request)



class ProfileSpider(CrawlSpider):
    name = 'pccprofile2'
    rules = [Rule(SgmlLinkExtractor(allow=(),restrict_xpaths=("//div[@class='table']")), callback='parse_item')]  

    def parse_item(self, response):
    hxs = HtmlXPathSelector(response)
    items = []
    item = Ppcprofile2Item()
    item ["name"] = hxs.select("//h1/text()").extract()
    item ["title"] = hxs.select("//div[@id='contentbody']//tr//td//text()").extract()
    items.append(item)

    #click next page      
    while True:
        next = response.findElement(By.linkText("下一页"))
        try:
            next.click()
        except:
            break

    return(items)

最佳答案

使用Downloader Middleware在使用 Scrapy 定期处理它们之前捕获需要 selenium 的页面:

The downloader middleware is a framework of hooks into Scrapy’s request/response processing. It’s a light, low-level system for globally altering Scrapy’s requests and responses.

这是一个使用 PhantomJS 的非常基本的例子:

from scrapy.http import HtmlResponse
from selenium import webdriver

class JSMiddleware(object):
    def process_request(self, request, spider):
        driver = webdriver.PhantomJS()
        driver.get(request.url)

        body = driver.page_source
        return HtmlResponse(driver.current_url, body=body, encoding='utf-8', request=request)

一旦你返回HtmlResponse(或者一个TextResponse，如果这是你真正想要的)，Scrapy 将停止处理下载器并进入蜘蛛的parse方法:

If it returns a Response object, Scrapy won’t bother calling any other process_request() or process_exception() methods, or the appropriate download function; it’ll return that response. The process_response() methods of installed middleware is always called on every response.

在这种情况下，您可以像通常使用 HTML 一样继续使用蜘蛛的 parse 方法，只是页面上的 JS 已经执行。

提示:由于 Downloader Middleware 的 process_request 方法接受蜘蛛作为参数，您可以在蜘蛛中添加一个条件来检查您是否需要处理 JS，这将让您使用完全相同的蜘蛛类处理 JS 和非 JS 页面。

关于python - 将 selenium 响应 url 传递给 scrapy，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/31174330/