我在 R 中使用 RCurl 尝试从网站下载数据，但我无法找到要使用的 URL。这是网站:

http://www.invescopowershares.com/products/holdings.aspx?ticker=PGX

看看在右上角，在显示的表格上方，有一个将数据下载为 .csv 文件的链接？我想知道是否有办法找到该 .csv 文件的常规 HTTP 地址，因为 RCurl 无法处理 Javascript 命令。

最佳答案

我会给你一个快速而肮脏的方法来获取数据。首先你可以使用 Fiddler2 http://www.fiddler2.com/fiddler2/检查浏览器发送的 POST。这导致以下 POST:

POST http://www.invescopowershares.com/products/holdings.aspx?ticker=PGX HTTP/1.1
Host: www.invescopowershares.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20100101 Firefox/13.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Referer: http://www.invescopowershares.com/products/holdings.aspx?ticker=PGX
Content-Type: application/x-www-form-urlencoded
Content-Length: 70669

__EVENTTARGET=ctl00%24MainPageLeft%24MainPageContent%24ExportHoldings1%24LinkButton1&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUKLTE1OTcxNjYzNw9kFgJmD2QWBAIDD2QWBAIDD2QWCAIBDw9kFgQeC2........

因此我们可以看到正在发布 3 个参数，即 __EVENTTARGET、__EVENTVALIDATION 和 __VIEWSTATE。

postForm 调用所需的表单是:

postForm(ftarget, "form name" = "aspnetForm", "method" = "POST", "action" = "holdings.aspx?ticker=PGX", "id" = "aspnetForm","__EVENTTARGET"=event.target,"__EVENTVALIDATION"=event.val,"__VIEWSTATE"=view.state)

现在是快速而肮脏的一点。我会打开浏览器并获取它收到的相关参数，如下所示:

library(rcom)
ie = comCreateObject('InternetExplorer.Application')
ie[["visible"]]=T # true for debugging
ie$Navigate2("http://www.invescopowershares.com/products/holdings.aspx?ticker=PGX")
while(comGetProperty(ie,"busy")||comGetProperty(ie,"ReadyState")<4){
Sys.sleep(1)
print(comGetProperty(ie,"ReadyState"))
}
myDoc<-comGetProperty(ie,"Document")
myPW<-comGetProperty(myDoc,"parentWindow")
comInvoke(myPW,"execScript","var dumVar1=theForm.__EVENTVALIDATION.value;var dumVar2=theForm.__VIEWSTATE.value;","JavaScript")
event.val<-myPW[["dumVar1"]]
view.state<-myPW[["dumVar2"]]
event.target<-"ctl00$MainPageLeft$MainPageContent$ExportHoldings1$LinkButton1"
ie$Quit()
ftarget<-"http://www.invescopowershares.com/products/holdings.aspx?ticker=PGX"
web.data<-postForm(ftarget, "form name" = "aspnetForm", "method" = "POST", "action" = "holdings.aspx?ticker=PGX", "id" = "aspnetForm","__EVENTTARGET"=event.target,"__EVENTVALIDATION"=event.val,"__VIEWSTATE"=view.state)
write(web.data[1],'temp.csv')
fin.data<-read.csv('temp.csv')


> fin.data[1,]
  ticker SecurityNum                      Name CouponRate maturitydate
1    PGX   949746879 WELLS FARGO & COMPANY PFD       0.08             
     rating  Shares PercentageOfFund PositionDate
1 BBB+/Baa3 2538656       0.04442112   06/11/2012

__EVENTVALIDATION、__VIEWSTATE 可能总是相同的，或者它们可能是 session cookie。您可能可以使用 RCurl 获取它们，但正如我所说，这是快速而肮脏的解决方案，我们只采用 Internet Explorer 提供的解决方案。注意事项:

1).这需要安装了 IE 的 Windows 才能使用 rcom 位。

2).如果您正在运行 ie9，您可能需要将 invescopowershares.com 添加到兼容性 View 设置(因为微软似乎已经阻止了 event.val<-mypw[["dumvar1"]] 类型的="" com="">

编辑(更新)

更详细地浏览了网站 __EVENTVALIDATION，__VIEWSTATE 在初始页面上被设置为 javascript 变量。我们可以像下面这样以一种快速而肮脏的方式解析它们，而无需调用浏览器。

dum<-getURL("http://www.invescopowershares.com/products/holdings.aspx?ticker=PGX")
event.target<-"ctl00$MainPageLeft$MainPageContent$ExportHoldings1$LinkButton1"
event.val<-unlist(strsplit(dum,"__EVENTVALIDATION\" value=\""))[2]
event.val<-unlist(strsplit(event.val,"\" />\r\n\r\n<script"))[1]
view.state<-unlist(strsplit(dum,"id=\"__VIEWSTATE\" value=\""))[2]
view.state<-unlist(strsplit(view.state,"\" />\r\n\r\n\r\n<script"))[1]
ftarget<-"http://www.invescopowershares.com/products/holdings.aspx?ticker=PGX"
web.data<-postForm(ftarget, "form name" = "aspnetForm", "method" = "POST", "action" = "holdings.aspx?ticker=PGX", "id" = "aspnetForm","__EVENTTARGET"=event.target,"__EVENTVALIDATION"=event.val,"__VIEWSTATE"=view.state)
write(web.data[1],'temp.csv')
fin.data<-read.csv('temp.csv')

以上应该跨平台工作。

关于javascript - 查找 Javascript 链接的网址，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/3426508/