草庐IT

java - 在 mac os 上从源代码编译 Hadoop Native 库

coder 2024-01-07 原文

我正在努力从源代码编译最新的 Hadoop，但有下一个异常(exception):

[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-pipes ---
[INFO]
[INFO] --- hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) @ hadoop-pipes ---
[INFO] Running cmake /Users/romeo/Repository/hadoop/hadoop-tools/hadoop-pipes/src -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles
[INFO] with extra environment variables {}
[WARNING] CMake Error at /usr/local/Cellar/cmake/3.7.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:138 (message):
[WARNING]   Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
[WARNING]   system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR)
[WARNING] Call Stack (most recent call first):
[WARNING]   /usr/local/Cellar/cmake/3.7.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
[WARNING]   /usr/local/Cellar/cmake/3.7.0/share/cmake/Modules/FindOpenSSL.cmake:385 (find_package_handle_standard_args)
[WARNING]   CMakeLists.txt:24 (find_package)
[WARNING]
[WARNING]
[WARNING] -- Configuring incomplete, errors occurred!
[WARNING] See also "/Users/romeo/Repository/hadoop/hadoop-tools/hadoop-pipes/target/native/CMakeFiles/CMakeOutput.log".
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.920 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  0.939 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.191 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.458 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.170 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.170 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  6.001 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  2.502 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  7.454 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  4.214 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 56.405 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  5.317 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  7.725 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.044 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 15.292 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 39.539 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  2.075 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 22.573 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.311 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.049 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.044 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [  8.825 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 22.424 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.030 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  5.927 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [  8.990 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  2.795 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  4.752 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  7.784 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 14.938 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  1.469 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  4.031 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  2.766 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  2.808 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [  2.918 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.041 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  2.676 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  2.113 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.043 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  3.960 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.029 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  6.991 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.138 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 14.384 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 12.110 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  3.040 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  6.127 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  4.264 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  3.597 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.715 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [  6.028 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.248 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  3.660 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  3.836 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  3.238 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.621 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  2.315 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  4.120 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.797 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.888 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  1.628 s]
[INFO] Apache Hadoop Pipes ................................ FAILURE [  0.533 s]
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Client ............................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:12 min
[INFO] Finished at: 2016-11-26T07:18:34+01:00
[INFO] Final Memory: 146M/857M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-pipes: CMake failed with error code 1 -> [Help 1]
[ERROR]

正如您从输出中看到的那样，由于 Unresolved 依赖性“Apache Hadoop Maven 插件”，它在“Apache Hadoop 管道”处失败。但是，Maven 插件已完成，状态为 SUCCESS:

[ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT -> [Help 1]
org.apache.maven.plugin.PluginResolutionException: Plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:128)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getPluginDescriptor(DefaultMavenPluginManager.java:179)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getMojoDescriptor(DefaultMavenPluginManager.java:284)
    at org.apache.maven.plugin.DefaultBuildPluginManager.getMojoDescriptor(DefaultBuildPluginManager.java:241)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.setupMojoExecution(DefaultLifecycleExecutionPlanCalculator.java:169)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.setupMojoExecutions(DefaultLifecycleExecutionPlanCalculator.java:155)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan(DefaultLifecycleExecutionPlanCalculator.java:131)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan(DefaultLifecycleExecutionPlanCalculator.java:145)
    at org.apache.maven.lifecycle.internal.builder.BuilderCommon.resolveBuildPlan(BuilderCommon.java:96)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:109)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:223)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:294)
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:124)
    ... 26 more
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:434)
    ... 30 more
[ERROR]

如有任何建议或提示，我们将不胜感激。谢谢。

更新 1:安装 OpenSSL 确实解决了主要问题:

[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Hadoop Pipes 3.0.0-alpha2-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT is missing, no dependency information available

更新 2:执行 mvn install 后得到新异常:

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-pipes: CMake failed with error code 1 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-pipes: CMake failed with error code 1
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)

最佳答案

安装hadoop-maven-plugin可以先通过mvn install来完成。 

$ mvn clean install -DskipTests
$ mvn package -Pdist -Pnative -Dtar -DskipTests

依赖项详细信息在 BUILDING.txt 中进行了描述.

或者如果你有docker环境，start-build-env.sh创建可以在其上构建 Hadoop 的 docker 容器。

$ cd hadoop
$ ./start-build-env.sh # Automatically login

由于所有必需的依赖项都已安装在该容器中，因此您可以在该容器中构建包括 native 在内的 Hadoop 包。

关于java - 在 mac os 上从源代码编译 Hadoop Native 库，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/40815773/

HadoopNativeApacheINFOjavamacosmavencmake

有关java - 在 mac os 上从源代码编译 Hadoop Native 库的更多相关文章

  1. ruby - 如何在 buildr 项目中使用 Ruby 代码？ - 2

    如何在buildr项目中使用Ruby？我在很多不同的项目中使用过Ruby、JRuby、Java和Clojure。我目前正在使用我的标准Ruby开发一个模拟应用程序，我想尝试使用Clojure后端(我确实喜欢功能代码)以及JRubygui和测试套件。我还可以看到在未来的不同项目中使用Scala作为后端。我想我要为我的项目尝试一下buildr(http://buildr.apache.org/)，但我注意到buildr似乎没有设置为在项目中使用JRuby代码本身!这看起来有点傻，因为该工具旨在统一通用的JVM语言并且是在ruby中构建的。除了将输出的jar包含在一个独特的、仅限ruby​​

  2. ruby-on-rails - Rails 源代码 : initialize hash in a weird way? - 2

    在rails源中:https://github.com/rails/rails/blob/master/activesupport/lib/active_support/lazy_load_hooks.rb可以看到以下内容@load_hooks=Hash.new{|h,k|h[k]=[]}在IRB中，它只是初始化一个空哈希。和做有什么区别@load_hooks=Hash.new 最佳答案 查看rubydocumentationforHashnew→new_hashclicktotogglesourcenew(obj)→new_has

  3. java - 等价于 Java 中的 Ruby Hash - 2

    我真的很习惯使用Ruby编写以下代码:my_hash={}my_hash['test']=1Java中对应的数据结构是什么？ 最佳答案 HashMapmap=newHashMap();map.put("test",1);我假设？ 关于java-等价于Java中的RubyHash，我们在StackOverflow上找到一个类似的问题： https://stackoverflow.com/questions/22737685/

  4. ruby-on-rails - 浏览 Ruby 源代码 - 2

    我的主要目标是能够完全理解我正在使用的库/gem。我尝试在Github上从头到尾阅读源代码，但这真的很难。我认为更有趣、更温和的踏脚石就是在使用时阅读每个库/gem方法的源代码。例如，我想知道RubyonRails中的redirect_to方法是如何工作的:如何查找redirect_to方法的源代码？我知道在pry中我可以执行类似show-methodmethod的操作，但我如何才能对Rails框架中的方法执行此操作？您对我如何更好地理解Gem及其API有什么建议吗？仅仅阅读源代码似乎真的很难，尤其是对于框架。谢谢! 最佳答案 Ru

  5. ruby - 模块嵌套代码风格偏好 - 2

    我的假设是moduleAmoduleBendend和moduleA::Bend是一样的。我能够从thisblog找到解决方案,thisSOthread和andthisSOthread.为什么以及什么时候应该更喜欢紧凑语法A::B而不是另一个，因为它显然有一个缺点？我有一种直觉，它可能与性能有关，因为在更多命名空间中查找常量需要更多计算。但是我无法通过对普通类进行基准测试来验证这一点。 最佳答案 这两种写作方法经常被混淆。首先要说的是，据我所知，没有可衡量的性能差异。(在下面的书面示例中不断查找)最明显的区别，可能也是最著名的，是你的

  6. ruby - 寻找通过阅读代码确定编程语言的ruby gem？ - 2

    几个月前，我读了一篇关于ruby​​gem的博客文章，它可以通过阅读代码本身来确定编程语言。对于我的生活，我不记得博客或gem的名称。谷歌搜索“ruby编程语言猜测”及其变体也无济于事。有人碰巧知道相关gem的名称吗？ 最佳答案 是这个吗:http://github.com/chrislo/sourceclassifier/tree/master 关于ruby-寻找通过阅读代码确定编程语言的rubygem？，我们在StackOverflow上找到一个类似的问题：

  7. java - 从 JRuby 调用 Java 类的问题 - 2

    我正在尝试使用boilerpipe来自JRuby。我看过guide从JRuby调用Java，并成功地将它与另一个Java包一起使用，但无法弄清楚为什么同样的东西不能用于boilerpipe。我正在尝试基本上从JRuby中执行与此Java等效的操作:URLurl=newURL("http://www.example.com/some-location/index.html");Stringtext=ArticleExtractor.INSTANCE.getText(url);在JRuby中试过这个:require'java'url=java.net.URL.new("http://www

  8. ruby - Net::HTTP 获取源代码和状态 - 2

    我目前正在使用以下方法获取页面的源代码:Net::HTTP.get(URI.parse(page.url))我还想获取HTTP状态，而无需发出第二个请求。有没有办法用另一种方法做到这一点？我一直在查看文档，但似乎找不到我要找的东西。 最佳答案 在我看来，除非您需要一些真正的低级访问或控制，否则最好使用Ruby的内置Open::URI模块:require'open-uri'io=open('http://www.example.org/')#=>#body=io.read[0,50]#=>"["200","OK"]io.base_ur

  9. ruby - Sinatra set cache_control to static files in public folder编译错误 - 2

    我不知道为什么，但是当我设置这个设置时它无法编译设置:static_cache_control，[:public，:max_age=>300]这是我得到的syntaxerror,unexpectedtASSOC,expecting']'(SyntaxError)set:static_cache_control,[:public,:max_age=>300]^我只想将“过期”header设置为css、javaascript和图像文件。谢谢。 最佳答案 我猜您使用的是Ruby1.8.7。Sinatra文档中显示的语法似乎是在Ruby1.

  10. java - 我的模型类或其他类中应该有逻辑吗 - 2

    我只想对我一直在思考的这个问题有其他意见，例如我有classuser_controller和classuserclassUserattr_accessor:name,:usernameendclassUserController//dosomethingaboutanythingaboutusersend问题是我的User类中是否应该有逻辑user=User.newuser.do_something(user1)oritshouldbeuser_controller=UserController.newuser_controller.do_something(user1,user2)我

随机推荐