我有一个 CSV 文件，其中包含以下方式的数据:

data_id,data_text,data_author
1,"here some text...",anurag
2,"Hi, i am apsc...",apsc
3,"i am living in "NYC"",another user

我正在执行以下步骤来加载正确的数据 方法 1.

temp = LOAD'filepath' USING PigStorage(',');

当我转储温度时，数据右移，因为第二条记录中有一个额外的逗号。

方法二: 使用新行作为分隔符加载数据

temp = LOAD'filepath' USING PigStorage('\n');

它在 1 个包中给我 1 个记录。

我再次尝试实现 RegEx 来打破困境

mydata = FOREACH data GENERATE FLATTEN(REGEX_EXTRACT_ALL('\\s*,\\s*,\\s*'));

这是抛出错误:

Pig 堆栈跟踪

ERROR 1045: Could not infer the matching function for org.apache.pig.builtin.REGEX_EXTRACT_ALL as multiple or none of them fit. Please use an explicit cast.

org.apache.pig.impl.logicalLayer.validators.TypeCheckerException: ERROR 1059: Problem while reconciling output schema of ForEach at org.apache.pig.newplan.logical.visitor.TypeCheckingRelVisitor.throwTypeCheckerException(TypeCheckingRelVisitor.java:142) at org.apache.pig.newplan.logical.visitor.TypeCheckingRelVisitor.visit(TypeCheckingRelVisitor.java:182) at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:76) at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) at org.apache.pig.PigServer$Graph.compile(PigServer.java:1635) at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1572) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1544) at org.apache.pig.PigServer.registerQuery(PigServer.java:516) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:991) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:412) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) at org.apache.pig.Main.run(Main.java:538) at org.apache.pig.Main.main(Main.java:157) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:622) at org.apache.hadoop.util.RunJar.main(RunJar.java:160) Caused by: org.apache.pig.impl.logicalLayer.validators.TypeCheckerException: ERROR 1045: Could not infer the matching function for org.apache.pig.builtin.REGEX_EXTRACT_ALL as multiple or none of them fit. Please use an explicit cast. at org.apache.pig.newplan.logical.visitor.TypeCheckingExpVisitor.visit(TypeCheckingExpVisitor.java:775) at org.apache.pig.newplan.logical.expression.UserFuncExpression.accept(UserFuncExpression.java:88) at org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) at org.apache.pig.newplan.logical.visitor.TypeCheckingRelVisitor.visitExpressionPlan(TypeCheckingRelVisitor.java:191) at org.apache.pig.newplan.logical.visitor.TypeCheckingRelVisitor.visit(TypeCheckingRelVisitor.java:157) at org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:246) at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52) at org.apache.pig.newplan.logical.visitor.TypeCheckingRelVisitor.visit(TypeCheckingRelVisitor.java:174)

... 19 more

请帮忙。

最佳答案

您可以使用 csvexcelstorage加载您的数据。您必须下载 piggybank.jar并将其注册到您的 pigscript 中。

REGISTER /path_to_jar/piggybank.jar;
DEFINE CSVExcelStorage org.apache.pig.piggybank.storage.CSVExcelStorage();

A = LOAD 'filepath/file.txt' USING CSVExcelStorage(',') AS (f1:int,f2:chararray,f3:chararray);
DUMP A;

关于hadoop - PIG 右移问题，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/44986858/