我在从 PIG 命令行执行 Hadoop 命令时遇到问题。命令和错误堆栈在下面

我的导师怀疑这是因为 HADDOP_HOME 和 PIG_CLASSPATH 不正确。我使用的是 HADOOP 2.8.0 版。

所以，最初我有 HADOOP_HOME 作为

HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/

然后我切换了以下设置:

HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/libexec/etc/hadoop

PIG_CLASSPATH 定义为 $HADOOP_HOME

我在 pig 中使用的命令:

A = LOAD '/Users/anarinsky/Downloads/loaddata1.txt';

B = MAPREDUCE '/Users/anarinsky/workspace/wordcount/target/wordcount-1.jar' STORE A INTO '/Users/anarinsky/Downloads/tempwrite2' LOAD  '/Users/anarinsky/Downloads/tempwrite2' AS (word:chararray, count:int) `com.systemskills.hadoop.wordcount.WordCountDriver /wordcountdata /Users/anarinsky/Downloads/pigoptdir`;

Pig 堆栈跟踪

ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias B
    at org.apache.pig.PigServer.openIterator(PigServer.java:1019)
    at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:747)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
    at org.apache.pig.Main.run(Main.java:564)
    at org.apache.pig.Main.main(Main.java:176)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias B
    at org.apache.pig.PigServer.storeEx(PigServer.java:1122)
    at org.apache.pig.PigServer.store(PigServer.java:1081)
    at org.apache.pig.PigServer.openIterator(PigServer.java:994)
    ... 13 more
Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:321)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:629)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:308)
    at org.apache.pig.PigServer.launchPlan(PigServer.java:1474)
    at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1459)
    at org.apache.pig.PigServer.storeEx(PigServer.java:1118)
    ... 15 more

最佳答案

亚历克斯! 不幸的是，它与具有相同结果的 Pig 路径(在我配置的 hadoop 集群上试过)无关。你得到的错误是指物理计划编译器在编译方法中有一个错误。因此，为了使您的尝试成功，您有两种可能性

1. 使用 hadoop 运行 native MR 作业，并在完成处理后在 pig 中生成结果

2. 编辑 pig 源代码并编译您自己的版本。你需要编辑 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler#compile方法和替换

   for (PhysicalOperator op : leaves) {
       if (!(op instanceof POStore)) {
           int errCode = 2025;
           String msg = "Expected leaf of reduce plan to " +
               "always be POStore. Found " + op.getClass().getSimpleName();
           throw new MRCompilerException(msg, errCode, PigException.BUG);
       }
   }
   

与

    for (PhysicalOperator op : leaves) {
        if (!(op instanceof POStore) && !(op instanceof PONative)) {
            int errCode = 2025;
            String msg = "Expected leaf of reduce plan to " +
                "always be POStore. Found " + op.getClass().getSimpleName();
            throw new MRCompilerException(msg, errCode, PigException.BUG);
        }
    }

关于java - APACHE HADOOP 版本 2.8.0 的 HADOOP_HOME 和 PIG_CLASSPATH 的正确值是多少？，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/43924904/