我正在使用 CDH5.3，我正在尝试编写一个 mapreduce 程序来扫描表并进行一些处理。我创建了一个扩展 TableMapper 的映射器，我得到的异常是:

java.io.FileNotFoundException: File does not exist: hdfs://localhost:54310/usr/local/hadoop-2.5-cdh-3.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1093)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:267)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:388)

但正如您在这里注意到的那样，它正在 hdfs 路径中搜索 protobuf-java-2.5.0.jar 但实际上它存在于本地路径中 -/usr/local/hadoop-2.5-cdh-3.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar ，我验证过。普通的 mapreduce 程序不会发生这种情况。只有当我使用 TableMapper 时才会发生此错误。

我的驱动代码如下:

   public class AppDriver  {

public static void main(String[] args) throws Exception{
 Configuration hbaseConfig = HBaseConfiguration.create();
    hbaseConfig.set("hbase.zookeeper.quorum", PropertiesUtil.getZookeperHostName());
    hbaseConfig.set("hbase.zookeeper.property.clientport", PropertiesUtil.getZookeperPortNum());

 Job job = Job.getInstance(hbaseConfig, "hbasemapreducejob");

    job.setJarByClass( AppDriver.class );

    // Create a scan
    Scan scan = new Scan();

    scan.setCaching(500);        // 1 is the default in Scan, which will be bad for MapReduce jobs
    scan.setCacheBlocks(false);    // don't set to true for MR jobs
    // scan.setStartRow(Bytes.toBytes(PropertiesUtil.getHbaseStartRowkey()));
    //  scan.setStopRow(Bytes.toBytes(PropertiesUtil.getHbaseStopRowkey()));

 TableMapReduceUtil.initTableMapperJob(PropertiesUtil.getHbaseTableName(),scan, ESportMapper.class, Text.class, RecordStatusVO.class, job);
    job.setReducerClass( ESportReducer.class );

    job.setNumReduceTasks(1);
    TableMapReduceUtil.addDependencyJars(job);

    // Write the results to a file in the output directory
    FileOutputFormat.setOutputPath( job, new Path( args[1] ));


   boolean b = job.waitForCompletion(true);
    if (!b) {
        throw new IOException("error with job!");
    }

}

我将属性文件作为 args[0] 。

一些更多的下划线信息:

i am using standalone CDH 5.3 in my local system and hbase 0.98.6 . my hbase is running on top of hdfs in sudo distributed mode .

我的 gradle.build 如下:

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'application' 
 // Basic Properties
 sourceCompatibility = 1.7
 targetCompatibility = '1.7'

 version = '3.0'
 mainClassName ="com.ESport.mapreduce.App.AppDriver"


 jar {
    manifest { 
     attributes "Main-Class": "$mainClassName"
  }  

 from {
    configurations.compile.collect { it.isDirectory() ? it :   zipTree(it) }
}

 zip64 true
}


repositories {
mavenCentral()
maven { url "http://clojars.org/repo" }
maven { url "http://repository.cloudera.com/artifactory/cloudera-  repos/" }
}

dependencies {

testCompile group: 'junit', name: 'junit', version: '4.+'

compile group: 'commons-collections', name: 'commons-collections', version: '3.2'
compile 'org.apache.storm:storm-core:0.9.4'
compile 'org.apache.commons:commons-compress:1.5'
compile 'org.elasticsearch:elasticsearch:1.7.1'

compile('org.apache.hadoop:hadoop-client:2.5.0-cdh5.3.0'){
    exclude group: 'org.slf4j'
}
compile('org.apache.hbase:hbase-client:0.98.6-cdh5.3.0') {

    exclude group: 'org.slf4j'
    exclude group: 'org.jruby'
    exclude group: 'jruby-complete'
    exclude group: 'org.codehaus.jackson'

}

compile 'org.apache.hbase:hbase-common:0.98.6-cdh5.3.0'
compile 'org.apache.hbase:hbase-server:0.98.6-cdh5.3.0'
compile 'org.apache.hbase:hbase-protocol:0.98.6-cdh5.3.0'

compile('com.thinkaurelius.titan:titan-core:0.5.2'){
    exclude group: 'org.slf4j'
}
compile('com.thinkaurelius.titan:titan-hbase:0.5.2'){
    exclude group: 'org.apache.hbase'
    exclude group: 'org.slf4j'
}
compile('com.tinkerpop.gremlin:gremlin-java:2.6.0'){
    exclude group: 'org.slf4j'
}
compile 'org.perf4j:perf4j:0.9.16'

compile 'com.fasterxml.jackson.core:jackson-core:2.5.3'
compile 'com.fasterxml.jackson.core:jackson-databind:2.5.3'
compile 'com.fasterxml.jackson.core:jackson-annotations:2.5.3'
compile 'com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.1.2'

我正在使用这个命令来运行 jar :

hadoop jar ESportingMapReduce-3.0.jar config.properties /myoutput

最佳答案

如果您尝试以伪分布式模式在 hbase 中进行设置，最可能的原因是将 hadoop 主页添加到 $PATH。
通过从 $PATH 中删除 hadoop home，您可以在伪分布式模式下启动 hbase。
有些人默认在.bashrc中添加hadoop home。
如果您将其添加到 .bashrc 中，请从中删除 hadoop home。

关于hadoop - 使用 TableMapper 时 HBase Mapreduce 依赖问题，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/34349720/