java - hadoop mapreduce 给出子错误

coder 2024-01-08 原文

我在 ubuntu 13.10 上使用 hadoop 1.2.1。我正在运行输入文件大小为 25GB 的排序问题。但是我收到错误:

 14/09/29 12:42:47 INFO mapred.JobClient:  map 51% reduce 17%
14/09/29 12:44:08 INFO mapred.JobClient: Task Id : attempt_201409291048_0003_m_000208_0, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

attempt_201409291048_0003_m_000208_0: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f4cfbad0000, 1683161088, 0) failed; error='Cannot allocate memory' (errno=12)
attempt_201409291048_0003_m_000208_0: #
attempt_201409291048_0003_m_000208_0: # There is insufficient memory for the Java Runtime Environment to continue.
attempt_201409291048_0003_m_000208_0: # Native memory allocation (malloc) failed to allocate 1683161088 bytes for committing reserved memory.
attempt_201409291048_0003_m_000208_0: # An error report file with more information is saved as:
attempt_201409291048_0003_m_000208_0: # /tmp/hadoop-hduser/mapred/local/taskTracker/hduser/jobcache/job_201409291048_0003/attempt_201409291048_0003_m_000208_0/work/hs_err_pid11760.log
14/09/29 12:44:10 INFO mapred.JobClient: Task Id : attempt_201409291048_0003_m_000209_0, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

attempt_201409291048_0003_m_000209_0: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f76efad0000, 1683161088, 0) failed; error='Cannot allocate memory' (errno=12)
attempt_201409291048_0003_m_000209_0: #
attempt_201409291048_0003_m_000209_0: # There is insufficient memory for the Java Runtime Environment to continue.
attempt_201409291048_0003_m_000209_0: # Native memory allocation (malloc) failed to allocate 1683161088 bytes for committing reserved memory.
attempt_201409291048_0003_m_000209_0: # An error report file with more information is saved as:
attempt_201409291048_0003_m_000209_0: # /tmp/hadoop-hduser/mapred/local/taskTracker/hduser/jobcache/job_201409291048_0003/attempt_201409291048_0003_m_000209_0/work/hs_err_pid11761.log
14/09/29 12:44:14 INFO mapred.JobClient: Task Id : attempt_201409291048_0003_m_000208_1, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

attempt_201409291048_0003_m_000208_1: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f0977ad0000, 1683161088, 0) failed; error='Cannot allocate memory' (errno=12)
attempt_201409291048_0003_m_000208_1: #
attempt_201409291048_0003_m_000208_1: # There is insufficient memory for the Java Runtime Environment to continue.
attempt_201409291048_0003_m_000208_1: # Native memory allocation (malloc) failed to allocate 1683161088 bytes for committing reserved memory.
attempt_201409291048_0003_m_000208_1: # An error report file with more information is saved as:
attempt_201409291048_0003_m_000208_1: # /tmp/hadoop-hduser/mapred/local/taskTracker/hduser/jobcache/job_201409291048_0003/attempt_201409291048_0003_m_000208_1/work/hs_err_pid11841.log
14/09/29 12:44:14 INFO mapred.JobClient: Task Id : attempt_201409291048_0003_m_000209_1, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

attempt_201409291048_0003_m_000209_1: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f76ebad0000, 1683161088, 0) failed; error='Cannot allocate memory' (errno=12)
attempt_201409291048_0003_m_000209_1: #
attempt_201409291048_0003_m_000209_1: # There is insufficient memory for the Java Runtime Environment to continue.
attempt_201409291048_0003_m_000209_1: # Native memory allocation (malloc) failed to allocate 1683161088 bytes for committing reserved memory.
attempt_201409291048_0003_m_000209_1: # An error report file with more information is saved as:
attempt_201409291048_0003_m_000209_1: # /tmp/hadoop-hduser/mapred/local/taskTracker/hduser/jobcache/job_201409291048_0003/attempt_201409291048_0003_m_000209_1/work/hs_err_pid11857.log
14/09/29 12:44:20 INFO mapred.JobClient: Task Id : attempt_201409291048_0003_m_000208_2, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

attempt_201409291048_0003_m_000208_2: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fdfdfad0000, 1683161088, 0) failed; error='Cannot allocate memory' (errno=12)
attempt_201409291048_0003_m_000208_2: #
attempt_201409291048_0003_m_000208_2: # There is insufficient memory for the Java Runtime Environment to continue.
attempt_201409291048_0003_m_000208_2: # Native memory allocation (malloc) failed to allocate 1683161088 bytes for committing reserved memory.
attempt_201409291048_0003_m_000208_2: # An error report file with more information is saved as:
attempt_201409291048_0003_m_000208_2: # /tmp/hadoop-hduser/mapred/local/taskTracker/hduser/jobcache/job_201409291048_0003/attempt_201409291048_0003_m_000208_2/work/hs_err_pid11922.log
14/09/29 12:44:22 INFO mapred.JobClient: Task Id : attempt_201409291048_0003_m_000209_2, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

attempt_201409291048_0003_m_000209_2: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f67ffad0000, 1683161088, 0) failed; error='Cannot allocate memory' (errno=12)
attempt_201409291048_0003_m_000209_2: #
attempt_201409291048_0003_m_000209_2: # There is insufficient memory for the Java Runtime Environment to continue.
attempt_201409291048_0003_m_000209_2: # Native memory allocation (malloc) failed to allocate 1683161088 bytes for committing reserved memory.
attempt_201409291048_0003_m_000209_2: # An error report file with more information is saved as:
attempt_201409291048_0003_m_000209_2: # /tmp/hadoop-hduser/mapred/local/taskTracker/hduser/jobcache/job_201409291048_0003/attempt_201409291048_0003_m_000209_2/work/hs_err_pid11938.log
14/09/29 12:44:30 INFO mapred.JobClient: Task Id : attempt_201409291048_0003_m_000402_0, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

attempt_201409291048_0003_m_000402_0: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f310fad0000, 1683161088, 0) failed; error='Cannot allocate memory' (errno=12)
attempt_201409291048_0003_m_000402_0: #
attempt_201409291048_0003_m_000402_0: # There is insufficient memory for the Java Runtime Environment to continue.
attempt_201409291048_0003_m_000402_0: # Native memory allocation (malloc) failed to allocate 1683161088 bytes for committing reserved memory.
attempt_201409291048_0003_m_000402_0: # An error report file with more information is saved as:
attempt_201409291048_0003_m_000402_0: # /tmp/hadoop-hduser/mapred/local/taskTracker/hduser/jobcache/job_201409291048_0003/attempt_201409291048_0003_m_000402_0/work/hs_err_pid12083.log
14/09/29 12:44:34 INFO mapred.JobClient: Task Id : attempt_201409291048_0003_m_000402_1, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

attempt_201409291048_0003_m_000402_1: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f356bad0000, 1683161088, 0) failed; error='Cannot allocate memory' (errno=12)
attempt_201409291048_0003_m_000402_1: #
attempt_201409291048_0003_m_000402_1: # There is insufficient memory for the Java Runtime Environment to continue.
attempt_201409291048_0003_m_000402_1: # Native memory allocation (malloc) failed to allocate 1683161088 bytes for committing reserved memory.
attempt_201409291048_0003_m_000402_1: # An error report file with more information is saved as:
attempt_201409291048_0003_m_000402_1: # /tmp/hadoop-hduser/mapred/local/taskTracker/hduser/jobcache/job_201409291048_0003/attempt_201409291048_0003_m_000402_1/work/hs_err_pid12102.log
14/09/29 12:44:40 INFO mapred.JobClient: Job complete: job_201409291048_0003
14/09/29 12:44:43 INFO mapred.JobClient: Counters: 24
14/09/29 12:44:43 INFO mapred.JobClient:   Job Counters 
14/09/29 12:44:43 INFO mapred.JobClient:     Launched reduce tasks=1
14/09/29 12:44:43 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=4441493
14/09/29 12:44:43 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/09/29 12:44:43 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/09/29 12:44:43 INFO mapred.JobClient:     Launched map tasks=216
14/09/29 12:44:43 INFO mapred.JobClient:     Data-local map tasks=216
14/09/29 12:44:43 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=2193941
14/09/29 12:44:43 INFO mapred.JobClient:     Failed map tasks=1
14/09/29 12:44:43 INFO mapred.JobClient:   File Input Format Counters 
14/09/29 12:44:43 INFO mapred.JobClient:     Bytes Read=13960068994
14/09/29 12:44:43 INFO mapred.JobClient:   FileSystemCounters
14/09/29 12:44:43 INFO mapred.JobClient:     HDFS_BYTES_READ=13962408717
14/09/29 12:44:43 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=13942458439
14/09/29 12:44:43 INFO mapred.JobClient:   Map-Reduce Framework
14/09/29 12:44:43 INFO mapred.JobClient:     Map output materialized bytes=13930872325
14/09/29 12:44:43 INFO mapred.JobClient:     Map input records=1323773
14/09/29 12:44:43 INFO mapred.JobClient:     Spilled Records=1323773
14/09/29 12:44:43 INFO mapred.JobClient:     Map output bytes=13923429356
14/09/29 12:44:43 INFO mapred.JobClient:     Total committed heap usage (bytes)=47269806080
14/09/29 12:44:43 INFO mapred.JobClient:     CPU time spent (ms)=866620
14/09/29 12:44:43 INFO mapred.JobClient:     Map input bytes=13958643740
14/09/29 12:44:43 INFO mapred.JobClient:     SPLIT_RAW_BYTES=22464
14/09/29 12:44:43 INFO mapred.JobClient:     Combine input records=0
14/09/29 12:44:43 INFO mapred.JobClient:     Combine output records=0
14/09/29 12:44:43 INFO mapred.JobClient:     Physical memory (bytes) snapshot=40872820736
14/09/29 12:44:43 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=11696959963136
14/09/29 12:44:43 INFO mapred.JobClient:     Map output records=1323773
14/09/29 12:44:43 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201409291048_0003_m_000208
java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
    at org.apache.hadoop.examples.Sort.run(Sort.java:176)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.examples.Sort.main(Sort.java:187)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

排序示例对 10GB 的输入文件工作得很好。我试图增加 JVM_ARGS 和 MaxPermSize。但问题仍然存在。任何建议将不胜感激。

最佳答案

此问题可能有多种原因，您需要检查是否达到作业数量限制？ 1) 日志目录空间不足，无法创建日志，

permissions issue. 2) ulimit threshold that causes insuffucient allocation of memory. 3) OOM on the child or unable to allocate the configured memory while spawning the child 4) Bug in the child args configuration in the mapred-site 5) Unable to write the temp outputs (due to space or permission issue) Hope it will help you

关于java - hadoop mapreduce 给出子错误，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/26105128/

有关java - hadoop mapreduce 给出子错误的更多相关文章

ruby-on-rails - Rails 常用字符串(用于通知和错误信息等) - 2
大约一年前，我决定确保每个包含非唯一文本的Flash通知都将从模块中的方法中获取文本。我这样做的最初原因是为了避免一遍又一遍地输入相同的字符串。如果我想更改措辞，我可以在一个地方轻松完成，而且一遍又一遍地重复同一件事而出现拼写错误的可能性也会降低。我最终得到的是这样的:moduleMessagesdefformat_error_messages(errors)errors.map{|attribute,message|"Error:#{attribute.to_s.titleize}#{message}."}enddeferror_message_could_not_find(obje
java - 等价于 Java 中的 Ruby Hash - 2
我真的很习惯使用Ruby编写以下代码:my_hash={}my_hash['test']=1Java中对应的数据结构是什么？最佳答案 HashMapmap=newHashMap();map.put("test",1);我假设？关于java-等价于Java中的RubyHash，我们在StackOverflow上找到一个类似的问题： https://stackoverflow.com/questions/22737685/
ruby-on-rails - 迷你测试错误 : "NameError: uninitialized constant" - 2
我遵循MichaelHartl的“RubyonRails教程:学习Web开发”，并创建了检查用户名和电子邮件长度有效性的测试(名称最多50个字符，电子邮件最多255个字符)。test/helpers/application_helper_test.rb的内容是:require'test_helper'classApplicationHelperTest在运行bundleexecraketest时，所有测试都通过了，但我看到以下消息在最后被标记为错误:ERROR["test_full_title_helper",ApplicationHelperTest,1.820016791]test
ruby-on-rails - 如何在 Rails View 上显示错误消息？ - 2
我是rails的新手，想在form字段上应用验证。myviewsnew.html.erb.....模拟.rbclassSimulation{:in=>1..25,:message=>'Therowmustbebetween1and25'}end模拟Controller.rbclassSimulationsController我想检查模型类中row字段的整数范围，如果不在范围内则返回错误信息。我可以检查上面代码的范围，但无法返回错误消息提前致谢最佳答案关键是您使用的是模型表单，一种显示ActiveRecord模型实例属性的表单。c
使用 ACL 调用 upload_file 时出现 Ruby S3 "Access Denied"错误 - 2
我正在尝试编写一个将文件上传到AWS并公开该文件的Ruby脚本。我做了以下事情:s3=Aws::S3::Resource.new(credentials:Aws::Credentials.new(KEY,SECRET),region:'us-west-2')obj=s3.bucket('stg-db').object('key')obj.upload_file(filename)这似乎工作正常，除了该文件不是公开可用的，而且我无法获得它的公共(public)URL。但是当我登录到S3时，我可以正常查看我的文件。为了使其公开可用，我将最后一行更改为obj.upload_file(file
ruby-on-rails - 错误 : Error installing pg: ERROR: Failed to build gem native extension - 2
我克隆了一个rails仓库，我现在正尝试捆绑安装背景:OSXElCapitanruby2.2.3p173(2015-08-18修订版51636)[x86_64-darwin15]rails-v在您的Gemfile中列出的或native可用的任何gem源中找不到gem'pg(>=0)ruby'。运行bundleinstall以安装缺少的gem。bundleinstallFetchinggemmetadatafromhttps://rubygems.org/............Fetchingversionmetadatafromhttps://rubygems.org/...Fe
ruby - ＃之间？ Cooper 的 *Beginning Ruby* 中的错误或异常 - 2
在Cooper的书BeginningRuby中，第166页有一个我无法重现的示例。classSongincludeComparableattr_accessor:lengthdef(other)@lengthother.lengthenddefinitialize(song_name,length)@song_name=song_name@length=lengthendenda=Song.new('Rockaroundtheclock',143)b=Song.new('BohemianRhapsody',544)c=Song.new('MinuteWaltz',60)a.betwee
ruby-on-rails - 每次我尝试部署时，我都会得到 - (gcloud.preview.app.deploy) 错误响应 : [4] DEADLINE_EXCEEDED - 2
我是Google云的新手，我正在尝试对其进行首次部署。我的第一个部署是RubyonRails项目。我基本上是在关注thisguideinthegoogleclouddocumentation.唯一的区别是我使用的是我自己的项目，而不是他们提供的“helloworld”项目。这是我的app.yaml文件runtime:customvm:trueentrypoint:bundleexecrackup-p8080-Eproductionconfig.ruresources:cpu:0.5memory_gb:1.3disk_size_gb:10当我转到我的项目目录并运行gcloudprevie
ruby-on-rails - Rails 5 Active Record 记录无效错误 - 2
我有两个Rails模型，即Invoice和Invoice_details。一个Invoice_details属于Invoice，一个Invoice有多个Invoice_details。我无法使用accepts_nested_attributes_forinInvoice通过Invoice模型保存Invoice_details。我收到以下错误:(0.2ms)BEGIN(0.2ms)ROLLBACKCompleted422UnprocessableEntityin25ms(ActiveRecord:4.0ms)ActiveRecord::RecordInvalid(Validationfa
arrays - 这是 Ruby 中 Array.fill 方法的错误吗？ - 2
这个问题在这里已经有了答案:Arraysmisbehaving(1个回答)关闭6年前。是否应该这样，即我误解了，还是错误？a=Array.new(3,Array.new(3))a[1].fill('g')=>[["g","g","g"],["g","g","g"],["g","g","g"]]它不应该导致:=>[[nil,nil,nil],["g","g","g"],[nil,nil,nil]]

java - hadoop mapreduce 给出子错误

有关java - hadoop mapreduce 给出子错误的更多相关文章

随机推荐