几年来，我一直在 Django 项目中使用 celery(python 中的异步任务队列)作为结果后端和代理 - 今天我尝试将后端/代理切换到 AWS 的 ElastiCache Redis，但是 celery worker 进入无限的启动和关闭循环，如下面的日志文件所示。我还从运行 celery 的同一台机器成功连接到 Redis 实例，并成功完成了一些基本的 SET 和 GET 操作。
我当前的设置是:
celery worker --concurrency=3 --queues=general --events --loglevel=DEBUG --beat --schedule=/tmp/celerybeat-schedule --pidfile=/tmp/celery.main.pid --config=celeryconfig
import os
from django.conf import settings
from celery import Celery, Task, chain
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings')
celery = Celery()
celery.config_from_object("celeryconfig")
celery.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')
BROKER_URL = "redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0"
CELERY_TIMEZONE = u'America/Los_Angeles'
CELERYD_HIJACK_ROOT_LOGGER = True
CELERY_IMPORTS = (
"app.tasks"
)
CELERYD_TASK_TIME_LIMIT = 35 # seconds
CELERYD_TASK_SOFT_TIME_LIMIT = 25 # seconds
DJANGO_SETTINGS_MODULE = "settings"
CELERY_RESULT_BACKEND = BROKER_URL
CELERY_ROUTES = {
'app.tasks.my_task': {'queue': 'general'}
}
-------------- celery@ip-10-20-21-9 v3.1.24 (Cipater)
---- **** -----
--- * *** * -- Linux-3.13.0-74-generic-x86_64-with-Ubuntu-14.04-trusty
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: default:0x7f7dc6d71650 (.default.Loader)
- ** ---------- .> transport: redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
- ** ---------- .> results: redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
- *** --- * --- .> concurrency: 3 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> general exchange=general(direct) key=general
[tasks]
. app.tasks.my_task
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
[2017-06-16 14:33:12,658: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
[2017-06-16 14:33:15,110: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2017-06-16 14:33:15,112: DEBUG/MainProcess] | Worker: Building graph...
[2017-06-16 14:33:15,113: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Queues (intra), Pool, Autoscaler, StateDB, Autoreloader, Beat, Consumer}
[2017-06-16 14:33:15,118: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2017-06-16 14:33:15,118: DEBUG/MainProcess] | Consumer: Building graph...
[2017-06-16 14:33:15,127: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Heart, Agent, Gossip, event loop}
[2017-06-16 14:33:15,137: DEBUG/MainProcess] | Worker: Starting Hub
[2017-06-16 14:33:15,137: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,137: DEBUG/MainProcess] | Worker: Starting Pool
[2017-06-16 14:33:15,287: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,288: DEBUG/MainProcess] | Worker: Starting Consumer
[2017-06-16 14:33:15,288: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-06-16 14:33:15,315: INFO/MainProcess] Connected to redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
[2017-06-16 14:33:15,315: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,316: DEBUG/MainProcess] | Consumer: Starting Events
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Hub...
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Pool...
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Consumer...
[2017-06-16 14:33:15,329: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2017-06-16 14:33:15,329: DEBUG/MainProcess] | Worker: Stopping Pool...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Worker: Stopping Hub...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2017-06-16 14:33:16,340: DEBUG/MainProcess] | Consumer: Shutdown Connection...
-------------- celery@ip-10-20-21-9 v3.1.24 (Cipater)
---- **** -----
--- * *** * -- Linux-3.13.0-74-generic-x86_64-with-Ubuntu-14.04-trusty
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: default:0x7fdc3b019650 (.default.Loader)
- ** ---------- .> transport: redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
- ** ---------- .> results: redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
- *** --- * --- .> concurrency: 3 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> general exchange=general(direct) key=general
[tasks]
. app.tasks.my_task
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
[2017-06-16 14:33:12,658: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
[2017-06-16 14:33:15,110: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2017-06-16 14:33:15,112: DEBUG/MainProcess] | Worker: Building graph...
[2017-06-16 14:33:15,113: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Queues (intra), Pool, Autoscaler, StateDB, Autoreloader, Beat, Consumer}
[2017-06-16 14:33:15,118: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2017-06-16 14:33:15,118: DEBUG/MainProcess] | Consumer: Building graph...
[2017-06-16 14:33:15,127: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Heart, Agent, Gossip, event loop}
[2017-06-16 14:33:15,137: DEBUG/MainProcess] | Worker: Starting Hub
[2017-06-16 14:33:15,137: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,137: DEBUG/MainProcess] | Worker: Starting Pool
[2017-06-16 14:33:15,287: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,288: DEBUG/MainProcess] | Worker: Starting Consumer
[2017-06-16 14:33:15,288: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-06-16 14:33:15,315: INFO/MainProcess] Connected to redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
[2017-06-16 14:33:15,315: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,316: DEBUG/MainProcess] | Consumer: Starting Events
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Hub...
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Pool...
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Consumer...
[2017-06-16 14:33:15,329: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2017-06-16 14:33:15,329: DEBUG/MainProcess] | Worker: Stopping Pool...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Worker: Stopping Hub...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2017-06-16 14:33:16,340: DEBUG/MainProcess] | Consumer: Shutdown Connection...
## And again...
最佳答案
问题的出现似乎是因为我尝试使用 Redis 集群作为中间件和后端，但开箱即用不受支持 - 在更基本的测试设置中，SETNX 命令是 celery 启动失败，因为 Redis 集群似乎不支持该命令。
我还需要做一些研究，看看是否可以使用 Redis 集群作为 Celery 后端和代理。
关于python - Celery worker 进入无限启动和关闭循环，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/44598537/
我需要在客户计算机上运行Ruby应用程序。通常需要几天才能完成(复制大备份文件)。问题是如果启用sleep，它会中断应用程序。否则，计算机将持续运行数周，直到我下次访问为止。有什么方法可以防止执行期间休眠并让Windows在执行后休眠吗？欢迎任何疯狂的想法;-) 最佳答案 Here建议使用SetThreadExecutionStateWinAPI函数，使应用程序能够通知系统它正在使用中，从而防止系统在应用程序运行时进入休眠状态或关闭显示。像这样的东西:require'Win32API'ES_AWAYMODE_REQUIRED=0x0
关闭。这个问题是opinion-based.它目前不接受答案。想要改进这个问题？更新问题，以便editingthispost可以用事实和引用来回答它.关闭4年前。Improvethisquestion我想在固定时间创建一系列低音和高音调的哔哔声。例如:在150毫秒时发出高音调的蜂鸣声在151毫秒时发出低音调的蜂鸣声200毫秒时发出低音调的蜂鸣声250毫秒的高音调蜂鸣声有没有办法在Ruby或Python中做到这一点？我真的不在乎输出编码是什么(.wav、.mp3、.ogg等等)，但我确实想创建一个输出文件。
我脑子里浮现出一些关于一种新编程语言的想法，所以我想我会尝试实现它。一位friend建议我尝试使用Treetop(Rubygem)来创建一个解析器。Treetop的文档很少，我以前从未做过这种事情。我的解析器表现得好像有一个无限循环，但没有堆栈跟踪；事实证明很难追踪到。有人可以指出入门级解析/AST指南的方向吗？我真的需要一些列出规则、常见用法等的东西来使用像Treetop这样的工具。我的语法分析器在GitHub上，以防有人希望帮助我改进它。class{initialize=lambda(name){receiver.name=name}greet=lambda{IO.puts("He
我有多个ActiveRecord子类Item的实例数组，我需要根据最早的事件循环打印。在这种情况下，我需要打印付款和维护日期，如下所示:ItemAmaintenancerequiredin5daysItemBpaymentrequiredin6daysItemApaymentrequiredin7daysItemBmaintenancerequiredin8days我目前有两个查询，用于查找maintenance和payment项目(非排他性查询)，并输出如下内容:paymentrequiredin...maintenancerequiredin...有什么方法可以改善上述(丑陋的)代
最近，当我启动我的Rails服务器时，我收到了一长串警告。虽然它不影响我的应用程序，但我想知道如何解决这些警告。我的估计是imagemagick以某种方式被调用了两次？当我在警告前后检查我的git日志时。我想知道如何解决这个问题。-bcrypt-ruby(3.1.2)-better_errors(1.0.1)+bcrypt(3.1.7)+bcrypt-ruby(3.1.5)-bcrypt(>=3.1.3)+better_errors(1.1.0)bcrypt和imagemagick有关系吗？/Users/rbchris/.rbenv/versions/2.0.0-p247/lib/ru
我收到这个错误:RuntimeError(自动加载常量Apps时检测到循环依赖当我使用多线程时。下面是我的代码。为什么会这样？我尝试多线程的原因是因为我正在编写一个HTML抓取应用程序。对Nokogiri::HTML(open())的调用是一个同步阻塞调用，需要1秒才能返回，我有100,000多个页面要访问，所以我试图运行多个线程来解决这个问题。有更好的方法吗？classToolsController0)app.website=array.join(',')putsapp.websiteelseapp.website="NONE"endapp.saveapps=Apps.order("
下面的代码在我第一次运行它时就可以正常工作:require'rubygems'require'spreadsheet'book=Spreadsheet.open'/Users/me/myruby/Mywks.xls'sheet=book.worksheet0row=sheet.row(1)putsrow[1]book.write'/Users/me/myruby/Mywks.xls'当我再次运行它时，我会收到更多消息，例如:/Library/Ruby/Gems/1.8/gems/spreadsheet-0.6.5.9/lib/spreadsheet/excel/reader.rb:11
这个问题在这里已经有了答案:关闭10年前。PossibleDuplicate:Pythonconditionalassignmentoperator对于这样一个简单的问题表示歉意，但是谷歌搜索||=并不是很有帮助；)Python中是否有与Ruby和Perl中的||=语句等效的语句？例如:foo="hey"foo||="what"#assignfooifit'sundefined#fooisstill"hey"bar||="yeah"#baris"yeah"另外，类似这样的东西的通用术语是什么？条件分配是我的第一个猜测，但Wikipediapage跟我想的不太一样。
什么是ruby的rack或python的Java的wsgi？还有一个路由库。 最佳答案 来自Python标准PEP333:Bycontrast,althoughJavahasjustasmanywebapplicationframeworksavailable,Java's"servlet"APImakesitpossibleforapplicationswrittenwithanyJavawebapplicationframeworktoruninanywebserverthatsupportstheservletAPI.ht
华为OD机试题本篇题目：明明的随机数题目输入描述输出描述：示例1输入输出说明代码编写思路最近更新的博客华为od2023|什么是华为od，od薪资待遇，od机试题清单华为OD机试真题大全，用Python解华为机试题|机试宝典【华为OD机试】全流程解析+经验分享,题型分享,防作弊指南华为o