草庐IT

python - 使用 tweepy 游标时,捕获容量过大错误的最佳做法是什么?

coder 2023-08-16 原文

我正在收集大量使用 Python 的 Tweepy 库的用户的信息。我已将 API 初始化如下

api = tweepy.API(auth,wait_on_rate_limit=True, wait_on_rate_limit_notify=True)

其中 auth 包含我的 token 。此代码可以很好地响应速率限制错误,但不适用于其他一些错误。例如,我有时会看到以下异常。

tweepy.error.TweepError: [{'message': 'Over capacity', 'code': 130}]

我可以使用 try except 来处理这个异常,但我想知道是否有一种方法可以在游标内处理这个异常,就像我处理速率限制异常一样。我看到像 retry_count 这样的参数,但我不确定它们是否适用于这种情况,因为它们似乎是为 HTTP 错误设计的。

最佳答案

I see parameters like retry_count, but I'm not sure if they will work for this case as they seem designed for HTTP errors.

嗯,according to the docs , 一些参数有一个 HTTP 代码对应,这表明它会起作用。 (整个列表在我的答案下方)

在整个列表中,你感兴趣的是

Over capacity 对应 HTTP 503。Twitter 暂时重载

因此,您需要在实例化 API 时提供一个 retry_errors 参数。这将告诉 API 重试哪些 HTTP 状态代码。一个例子:

retry_count=10, retry_delay=5, retry_errors=set([503])


整个代码列表:

+------+--------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Code |                                               Text                                               |                                                                                                                                                                                       Description                                                                                                                                                                                        |
+------+--------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|    3 | Invalid coordinates                                                                              | Corresponds with HTTP 400. The coordinates provided as parameters were not valid for the request.                                                                                                                                                                                                                                                                                        |
|   13 | No location associated with the specified IP address                                             | Corresponds with HTTP 404. It was not possible to derive a location for the IP address provided as a parameter on the geo search request.                                                                                                                                                                                                                                                |
|   17 | No user matches for specified terms                                                              | Corresponds with HTTP 404. It was not possible to find a user profile matching the parameters specified.                                                                                                                                                                                                                                                                                 |
|   32 | Could not authenticate you                                                                       | Corresponds with HTTP 401. There was an issue with the authentication data for the request.                                                                                                                                                                                                                                                                                              |
|   34 | Sorry, that page does not exist                                                                  | Corresponds with HTTP 404. The specified resource was not found.                                                                                                                                                                                                                                                                                                                         |
|   36 | You cannot report yourself for spam.                                                             | Corresponds with HTTP 403. You cannot use your own user ID in a report spam call.                                                                                                                                                                                                                                                                                                        |
|   44 | attachment_url parameter is invalid                                                              | Corresponds with HTTP 400. The URL value provided is not a URL that can be attached to this Tweet.                                                                                                                                                                                                                                                                                       |
|   50 | User not found.                                                                                  | Corresponds with HTTP 404. The user is not found.                                                                                                                                                                                                                                                                                                                                        |
|   63 | User has been suspended.                                                                         | Corresponds with HTTP 403 The user account has been suspended and information cannot be retrieved.                                                                                                                                                                                                                                                                                       |
|   64 | Your account is suspended and is not permitted to access this feature                            | Corresponds with HTTP 403. The access token being used belongs to a suspended user.                                                                                                                                                                                                                                                                                                      |
|   68 | The Twitter REST API v1 is no longer active. Please migrate to API v1.1.                         | Corresponds to a HTTP request to a retired v1-era URL.                                                                                                                                                                                                                                                                                                                                   |
|   87 | Client is not permitted to perform this action.                                                  | Corresponds with HTTP 403. The endpoint called is not a permitted URL.                                                                                                                                                                                                                                                                                                                   |
|   93 | This application is not allowed to access or delete your direct messages                         | Corresponds with HTTP 403. The OAuth token does not provide access to Direct Messages.                                                                                                                                                                                                                                                                                                   |
|   99 | Unable to verify your credentials.                                                               | Corresponds with HTTP 403. The OAuth credentials cannot be validated. Check that the token is still valid.                                                                                                                                                                                                                                                                               |
|  120 | Account update failed: value is too long (maximum is nn characters)                              | Corresponds with HTTP 403. Thrown when one of the values passed to the update_profile.json endpoint exceeds the maximum value currently permitted for that field. The error message will specify the allowable maximum number of nn characters.                                                                                                                                          |
|  130 | Over capacity                                                                                    | Corresponds with HTTP 503. Twitter is temporarily over capacity.                                                                                                                                                                                                                                                                                                                         |
|  131 | Internal error                                                                                   | Corresponds with HTTP 500. An unknown internal error occurred.                                                                                                                                                                                                                                                                                                                           |
|  135 | Could not authenticate you                                                                       | Corresponds with HTTP 401. Timestamp out of bounds (often caused by a clock drift when authenticating - check your system clock)                                                                                                                                                                                                                                                         |
|  144 | No status found with that ID.                                                                    | Corresponds with HTTP 404. The requested Tweet ID is not found (if it existed, it was probably deleted)                                                                                                                                                                                                                                                                                  |
|  150 | You cannot send messages to users who are not following you.                                     | Corresponds with HTTP 403. Sending a Direct Message failed.                                                                                                                                                                                                                                                                                                                              |
|  151 | There was an error sending your message: reason                                                  | Corresponds with HTTP 403. Sending a Direct Message failed. The reason value will provide more information.                                                                                                                                                                                                                                                                              |
|  160 | You've already requested to follow user                                                          | Corresponds with HTTP 403. This was a duplicated follow request and a previous request was not yet acknowleged.                                                                                                                                                                                                                                                                          |
|  161 | You are unable to follow more people at this time                                                | Corresponds with HTTP 403. Thrown when a user cannot follow another user due to some kind of limit                                                                                                                                                                                                                                                                                       |
|  179 | Sorry, you are not authorized to see this status                                                 | Corresponds with HTTP 403. Thrown when a Tweet cannot be viewed by the authenticating user, usually due to the Tweet’s author having protected their Tweets.                                                                                                                                                                                                                             |
|  185 | User is over daily status update limit                                                           | Corresponds with HTTP 403. Thrown when a Tweet cannot be posted due to the user having no allowance remaining to post. Despite the text in the error message indicating that this error is only thrown when a daily limit is reached, this error will be thrown whenever a posting limitation has been reached. Posting allowances have roaming windows of time of unspecified duration. |
|  186 | Tweet needs to be a bit shorter.                                                                 | Corresponds with HTTP 403. The status text is too long.                                                                                                                                                                                                                                                                                                                                  |
|  205 | You are over the limit for spam reports.                                                         | Corresponds with HTTP 403. The account limit for reporting spam has been reached. Try again later.                                                                                                                                                                                                                                                                                       |
|  215 | Bad authentication data                                                                          | Corresponds with HTTP 400. The method requires authentication but it was not presented or was wholly invalid.                                                                                                                                                                                                                                                                            |
|  220 | Your credentials do not allow access to this resource.                                           | Corresponds with HTTP 403. The authentication token in use is restricted and cannot access the requested resource.                                                                                                                                                                                                                                                                       |
|  251 | This endpoint has been retired and should not be used.                                           | Corresponds to a HTTP request to a retired URL.                                                                                                                                                                                                                                                                                                                                          |
|  261 | Application cannot perform write actions.                                                        | Corresponds with HTTP 403. Thrown when the application is restricted from POST, PUT, or DELETE actions. Check the information on your application dashboard. See How to appeal application suspension and other disciplinary actions.                                                                                                                                                    |
|  271 | You can’t mute yourself.                                                                         | Corresponds with HTTP 403. The authenticated user account cannot mute itself.                                                                                                                                                                                                                                                                                                            |
|  272 | You are not muting the specified user.                                                           | Corresponds with HTTP 403. The authenticated user account is not muting the account a call is attempting to unmute.                                                                                                                                                                                                                                                                      |
|  323 | Animated GIFs are not allowed when uploading multiple images.                                    | Corresponds with HTTP 400. Only one animated GIF is allowed to be attached to a single Tweet.                                                                                                                                                                                                                                                                                            |
|  324 | The validation of media ids failed.                                                              | Corresponds with HTTP 400. There was a problem with the media ID submitted with the Tweet.                                                                                                                                                                                                                                                                                               |
|  325 | A media id was not found.                                                                        | Corresponds with HTTP 400. The media ID attached to the Tweet was not found.                                                                                                                                                                                                                                                                                                             |
|  326 | To protect our users from spam and other malicious activity, this account is temporarily locked. | Corresponds with HTTP 403. The user should log in to https://twitter.com to unlock their account before the user token can be used.                                                                                                                                                                                                                                                      |
|  327 | You have already retweeted this Tweet                                                            | Corresponds with HTTP 403. The user cannot retweet the same Tweet more than once.                                                                                                                                                                                                                                                                                                        |
|  354 | The text of your direct message is over the max character limit.                                 | Corresponds with HTTP 403. The message size exceeds the number of characters permitted in a Direct Message.                                                                                                                                                                                                                                                                              |
|  385 | You attempted to reply to a Tweet that is deleted or not visible to you.                         | Corresponds with HTTP 403. A reply can only be sent with reference to an existing public Tweet.                                                                                                                                                                                                                                                                                          |
|  386 | The Tweet exceeds the number of allowed attachment types.                                        | Corresponds with HTTP 403. A Tweet is limited to a single attachment resource (media, Quote Tweet, etc.)                                                                                                                                                                                                                                                                                 |
+------+--------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

关于python - 使用 tweepy 游标时,捕获容量过大错误的最佳做法是什么?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/48117126/

有关python - 使用 tweepy 游标时,捕获容量过大错误的最佳做法是什么?的更多相关文章

  1. ruby - 如何使用 Nokogiri 的 xpath 和 at_xpath 方法 - 2

    我正在学习如何使用Nokogiri,根据这段代码我遇到了一些问题:require'rubygems'require'mechanize'post_agent=WWW::Mechanize.newpost_page=post_agent.get('http://www.vbulletin.org/forum/showthread.php?t=230708')puts"\nabsolutepathwithtbodygivesnil"putspost_page.parser.xpath('/html/body/div/div/div/div/div/table/tbody/tr/td/div

  2. ruby - 使用 RubyZip 生成 ZIP 文件时设置压缩级别 - 2

    我有一个Ruby程序,它使用rubyzip压缩XML文件的目录树。gem。我的问题是文件开始变得很重,我想提高压缩级别,因为压缩时间不是问题。我在rubyzipdocumentation中找不到一种为创建的ZIP文件指定压缩级别的方法。有人知道如何更改此设置吗?是否有另一个允许指定压缩级别的Ruby库? 最佳答案 这是我通过查看ruby​​zip内部创建的代码。level=Zlib::BEST_COMPRESSIONZip::ZipOutputStream.open(zip_file)do|zip|Dir.glob("**/*")d

  3. ruby - 为什么我可以在 Ruby 中使用 Object#send 访问私有(private)/ protected 方法? - 2

    类classAprivatedeffooputs:fooendpublicdefbarputs:barendprivatedefzimputs:zimendprotecteddefdibputs:dibendendA的实例a=A.new测试a.foorescueputs:faila.barrescueputs:faila.zimrescueputs:faila.dibrescueputs:faila.gazrescueputs:fail测试输出failbarfailfailfail.发送测试[:foo,:bar,:zim,:dib,:gaz].each{|m|a.send(m)resc

  4. ruby-on-rails - 使用 Ruby on Rails 进行自动化测试 - 最佳实践 - 2

    很好奇,就使用ruby​​onrails自动化单元测试而言,你们正在做什么?您是否创建了一个脚本来在cron中运行rake作业并将结果邮寄给您?git中的预提交Hook?只是手动调用?我完全理解测试,但想知道在错误发生之前捕获错误的最佳实践是什么。让我们理所当然地认为测试本身是完美无缺的,并且可以正常工作。下一步是什么以确保他们在正确的时间将可能有害的结果传达给您? 最佳答案 不确定您到底想听什么,但是有几个级别的自动代码库控制:在处理某项功能时,您可以使用类似autotest的内容获得关于哪些有效,哪些无效的即时反馈。要确保您的提

  5. ruby - 在 Ruby 中使用匿名模块 - 2

    假设我做了一个模块如下:m=Module.newdoclassCendend三个问题:除了对m的引用之外,还有什么方法可以访问C和m中的其他内容?我可以在创建匿名模块后为其命名吗(就像我输入“module...”一样)?如何在使用完匿名模块后将其删除,使其定义的常量不再存在? 最佳答案 三个答案:是的,使用ObjectSpace.此代码使c引用你的类(class)C不引用m:c=nilObjectSpace.each_object{|obj|c=objif(Class===objandobj.name=~/::C$/)}当然这取决于

  6. ruby - 使用 ruby​​ 和 savon 的 SOAP 服务 - 2

    我正在尝试使用ruby​​和Savon来使用网络服务。测试服务为http://www.webservicex.net/WS/WSDetails.aspx?WSID=9&CATID=2require'rubygems'require'savon'client=Savon::Client.new"http://www.webservicex.net/stockquote.asmx?WSDL"client.get_quotedo|soap|soap.body={:symbol=>"AAPL"}end返回SOAP异常。检查soap信封,在我看来soap请求没有正确的命名空间。任何人都可以建议我

  7. python - 如何使用 Ruby 或 Python 创建一系列高音调和低音调的蜂鸣声? - 2

    关闭。这个问题是opinion-based.它目前不接受答案。想要改进这个问题?更新问题,以便editingthispost可以用事实和引用来回答它.关闭4年前。Improvethisquestion我想在固定时间创建一系列低音和高音调的哔哔声。例如:在150毫秒时发出高音调的蜂鸣声在151毫秒时发出低音调的蜂鸣声200毫秒时发出低音调的蜂鸣声250毫秒的高音调蜂鸣声有没有办法在Ruby或Python中做到这一点?我真的不在乎输出编码是什么(.wav、.mp3、.ogg等等),但我确实想创建一个输出文件。

  8. ruby-on-rails - Rails - 子类化模型的设计模式是什么? - 2

    我有一个模型:classItem项目有一个属性“商店”基于存储的值,我希望Item对象对特定方法具有不同的行为。Rails中是否有针对此的通用设计模式?如果方法中没有大的if-else语句,这是如何干净利落地完成的? 最佳答案 通常通过Single-TableInheritance. 关于ruby-on-rails-Rails-子类化模型的设计模式是什么?,我们在StackOverflow上找到一个类似的问题: https://stackoverflow.co

  9. ruby-on-rails - 'compass watch' 是如何工作的/它是如何与 rails 一起使用的 - 2

    我在我的项目目录中完成了compasscreate.和compassinitrails。几个问题:我已将我的.sass文件放在public/stylesheets中。这是放置它们的正确位置吗?当我运行compasswatch时,它不会自动编译这些.sass文件。我必须手动指定文件:compasswatchpublic/stylesheets/myfile.sass等。如何让它自动运行?文件ie.css、print.css和screen.css已放在stylesheets/compiled。如何在编译后不让它们重新出现的情况下删除它们?我自己编译的.sass文件编译成compiled/t

  10. ruby - 使用 ruby​​ 将 HTML 转换为纯文本并维护结构/格式 - 2

    我想将html转换为纯文本。不过,我不想只删除标签,我想智能地保留尽可能多的格式。为插入换行符标签,检测段落并格式化它们等。输入非常简单,通常是格式良好的html(不是整个文档,只是一堆内容,通常没有anchor或图像)。我可以将几个正则表达式放在一起,让我达到80%,但我认为可能有一些现有的解决方案更智能。 最佳答案 首先,不要尝试为此使用正则表达式。很有可能你会想出一个脆弱/脆弱的解决方案,它会随着HTML的变化而崩溃,或者很难管理和维护。您可以使用Nokogiri快速解析HTML并提取文本:require'nokogiri'h

随机推荐