我通过brew install hadoop安装hadoop，然后使用pip install pyarrow作为客户端
client = pa.hdfs.connect('localhost', 9000, driver='libhdfs')
for x in client.ls('/'):
print(x)
获取错误
Traceback (most recent call last):
File "/Users/wyx/project/py3.7aio/hdfs/list_dir.py", line 13, in <module>
client = pa.hdfs.connect('localhost', 9000, )
File "/Users/wyx/project/py3.7aio/.env/lib/python3.6/site-packages/pyarrow/hdfs.py", line 207, in connect
extra_conf=extra_conf)
File "/Users/wyx/project/py3.7aio/.env/lib/python3.6/site-packages/pyarrow/hdfs.py", line 38, in __init__
self._connect(host, port, user, kerb_ticket, driver, extra_conf)
File "pyarrow/io-hdfs.pxi", line 89, in pyarrow.lib.HadoopFileSystem._connect
File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Unable to load libhdfs
如何获取libhdfs或libhdfs3？
我认为这可能与本地库有关，因为
➜ hadoop-rel-release-3.1.1 hadoop checknative -a
2019-02-21 15:46:20,805 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop: false
zlib: false
zstd : false
snappy: false
lz4: false
bzip2: false
openssl: false
ISA-L: false
2019-02-21 15:46:20,934 INFO util.ExitUtil: Exiting with status 1: ExitException
在我 brew install gcc autoconf automake libtool cmake snappy gzip bzip2 protobuf@2.5 zlib openssl 之后
下载 hadoop-rel-release-3.1.1 进行构建。
[INFO] Compiling 3 source files to /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) @ hadoop-yarn-server-nodemanager ---
[DEBUG] Configuring mojo org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile from plugin realm ClassRealm[plugin>org.apache.hadoop:hadoop-maven-plugins:3.1.1, parent: sun.misc.Launcher$AppClassLoader@55f96302]
[DEBUG] Configuring mojo 'org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile' with basic configurator -->
[DEBUG] (f) env = {CFLAGS=null}
[DEBUG] (f) output = /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native
[DEBUG] (f) source = /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src
[DEBUG] (f) vars = {HADOOP_CONF_DIR=../etc/hadoop, JVM_ARCH_DATA_MODEL=64}
[DEBUG] -- end configuration --
[INFO] Running cmake /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -DHADOOP_CONF_DIR=../etc/hadoop -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles
[INFO] with extra environment variables {
CFLAGS = ''
}
[INFO] Running make -j 8 VERBOSE=1
[WARNING] /usr/local/Cellar/cmake/3.13.4/bin/cmake -S/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -B/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native --check-build-system CMakeFiles/Makefile.cmake 0
[WARNING] /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_progress_start /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/progress.marks
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/Makefile2 all
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container.dir/build.make CMakeFiles/container.dir/depend
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/gtest.dir/build.make CMakeFiles/gtest.dir/depend
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/gtest.dir/DependInfo.cmake --color=
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/container.dir/DependInfo.cmake --color=
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/gtest.dir/build.make CMakeFiles/gtest.dir/build
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container.dir/build.make CMakeFiles/container.dir/build
[WARNING] make[2]: Nothing to be done for `CMakeFiles/gtest.dir/build'.
[WARNING] make[2]: Nothing to be done for `CMakeFiles/container.dir/build'.
[WARNING] [ 6%] Built target gtest
[WARNING] [ 46%] Built target container
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/cetest.dir/build.make CMakeFiles/cetest.dir/depend
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/test-container-executor.dir/build.make CMakeFiles/test-container-executor.dir/depend
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container-executor.dir/build.make CMakeFiles/container-executor.dir/depend
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/test-container-executor.dir/DependInfo.cmake --color=
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/container-executor.dir/DependInfo.cmake --color=
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/cetest.dir/DependInfo.cmake --color=
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container-executor.dir/build.make CMakeFiles/container-executor.dir/build
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/test-container-executor.dir/build.make CMakeFiles/test-container-executor.dir/build
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/cetest.dir/build.make CMakeFiles/cetest.dir/build
[WARNING] make[2]: Nothing to be done for `CMakeFiles/container-executor.dir/build'.
[WARNING] make[2]: Nothing to be done for `CMakeFiles/test-container-executor.dir/build'.
[WARNING] [ 60%] Built target container-executor
[WARNING] [ 60%] Built target test-container-executor
[WARNING] [ 63%] Building CXX object CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o
[WARNING] /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -I/System/Library/Frameworks -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl -isystem /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/../../../../../hadoop-common-project/hadoop-common/src/main/native/gtest/include -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -o CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o -c /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc
[WARNING] In file included from /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:56:13: warning: comparison of array 'args->data' equal to a null pointer is always false [-Wtautological-pointer-compare]
[WARNING] if (args->data == NULL || args->length >= DOCKER_ARG_MAX) {
[WARNING] ~~~~~~^~~~ ~~~~
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:63:13: warning: comparison of array 'args->data' not equal to a null pointer is always true [-Wtautological-pointer-compare]
[WARNING] if (args->data != NULL) {
[WARNING] ~~~~~~^~~~ ~~~~
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1227:12: error: no matching function for call to 'getgrouplist'
[WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
[WARNING] ^~~~~~~~~~~~
[WARNING] /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/unistd.h:653:6: note: candidate function not viable: no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd argument
[WARNING] int getgrouplist(const char *, int, int *, int *);
[WARNING] ^
[WARNING] In file included from /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1234:9: error: no matching function for call to 'getgrouplist'
[WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
[WARNING] ^~~~~~~~~~~~
[WARNING] /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/unistd.h:653:6: note: candidate function not viable: no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd argument
[WARNING] int getgrouplist(const char *, int, int *, int *);
[WARNING] ^
[WARNING] 2 warnings and 2 errors generated.
[WARNING] make[2]: *** [CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o] Error 1
[WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
[WARNING] make: *** [all] Error 2
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.1.1:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.115 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 1.784 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.991 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.055 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.579 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.898 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.680 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.903 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.898 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.586 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 47.036 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 4.246 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 5.686 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.385 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 16.175 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 36.668 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 3.188 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 6.474 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.781 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 13.786 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.317 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.291 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 8.560 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 29.751 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 4.885 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.354 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 8.905 s]
[INFO] Apache Hadoop YARN NodeManager ..................... FAILURE [ 2.680 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service ................ SKIPPED
[INFO] Apache Hadoop YARN ResourceManager ................. SKIPPED
[INFO] Apache Hadoop YARN Server Tests .................... SKIPPED
[INFO] Apache Hadoop YARN Client .......................... SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager .............. SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SKIPPED
[INFO] Apache Hadoop YARN Router .......................... SKIPPED
[INFO] Apache Hadoop YARN Applications .................... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ................ SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SKIPPED
[INFO] Apache Hadoop MapReduce Client ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Core ....................... SKIPPED
[INFO] Apache Hadoop MapReduce Common ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle .................... SKIPPED
[INFO] Apache Hadoop MapReduce App ........................ SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer .............. SKIPPED
[INFO] Apache Hadoop MapReduce JobClient .................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop YARN Services ........................ SKIPPED
[INFO] Apache Hadoop YARN Services Core ................... SKIPPED
[INFO] Apache Hadoop YARN Services API .................... SKIPPED
[INFO] Apache Hadoop YARN Site ............................ SKIPPED
[INFO] Apache Hadoop YARN UI .............................. SKIPPED
[INFO] Apache Hadoop YARN Project ......................... SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask ................. SKIPPED
[INFO] Apache Hadoop MapReduce Uploader ................... SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] Apache Hadoop MapReduce ............................ SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Archive Logs ......................... SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Client Aggregator .................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Resource Estimator Service ........... SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Image Generation Tool ................ SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:35 min
[INFO] Finished at: 2019-02-21T15:31:33+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with error code 2 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with error code 2
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with error code 2
at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake (CompileMojo.java:229)
at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute (CompileMojo.java:98)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-yarn-server-nodemanager
最佳答案
同样的问题，如果你仔细阅读你的编译日志，你会看到
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1227:12: error: no matching function for call to 'getgrouplist'
[WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
[WARNING] ^~~~~~~~~~~~
[WARNING] /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/unistd.h:653:6: note: candidate function not viable: no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd argument
[WARNING] int getgrouplist(const char *, int, int *, int *);
用谷歌搜索并发现一个问题 https://issues.apache.org/jira/browse/YARN-8622这里
以下更改对我来说效果很好:
source file: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c
line 1227:
- int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
+ int rc = getgrouplist(user, pw->pw_gid, (int *)groups, &ngroups);
line 1234:
- if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
+ if (getgrouplist(user, pw->pw_gid, (int *)groups, &ngroups) == -1) {
关于hadoop - 在 osx 中构建 hadoop 3.1.1 以获取 native 库，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/54801924/
我的最终目标是安装当前版本的RubyonRails。我在OSXMountainLion上运行。到目前为止，这是我的过程:已安装的RVM$\curl-Lhttps://get.rvm.io|bash-sstable检查已知(我假设已批准)安装$rvmlistknown我看到当前的稳定版本可用[ruby-]2.0.0[-p247]输入命令安装$rvminstall2.0.0-p247注意:我也试过这些安装命令$rvminstallruby-2.0.0-p247$rvminstallruby=2.0.0-p247我很快就无处可去了。结果:$rvminstall2.0.0-p247Search
有没有办法在这个简单的get方法中添加超时选项？我正在使用法拉第3.3。Faraday.get(url)四处寻找，我只能先发起连接后应用超时选项，然后应用超时选项。或者有什么简单的方法？这就是我现在正在做的:conn=Faraday.newresponse=conn.getdo|req|req.urlurlreq.options.timeout=2#2secondsend 最佳答案 试试这个:conn=Faraday.newdo|conn|conn.options.timeout=20endresponse=conn.get(url
我克隆了一个rails仓库，我现在正尝试捆绑安装背景:OSXElCapitanruby2.2.3p173(2015-08-18修订版51636)[x86_64-darwin15]rails-v在您的Gemfile中列出的或native可用的任何gem源中找不到gem'pg(>=0)ruby'。运行bundleinstall以安装缺少的gem。bundleinstallFetchinggemmetadatafromhttps://rubygems.org/............Fetchingversionmetadatafromhttps://rubygems.org/...Fe
我有一个存储主机名的Ruby数组server_names。如果我打印出来，它看起来像这样:["hostname.abc.com","hostname2.abc.com","hostname3.abc.com"]相当标准。我想要做的是获取这些服务器的IP(可能将它们存储在另一个变量中)。看起来IPSocket类可以做到这一点，但我不确定如何使用IPSocket类遍历它。如果它只是尝试像这样打印出IP:server_names.eachdo|name|IPSocket::getaddress(name)pnameend它提示我没有提供服务器名称。这是语法问题还是我没有正确使用类？输出:ge
我想获取模块中定义的所有常量的值:moduleLettersA='apple'.freezeB='boy'.freezeendconstants给了我常量的名字:Letters.constants(false)#=>[:A,:B]如何获取它们的值的数组，即["apple","boy"]？ 最佳答案 为了做到这一点，请使用mapLetters.constants(false).map&Letters.method(:const_get)这将返回["a","b"]第二种方式:Letters.constants(false).map{|c
我安装了ruby版本管理器，并将RVM安装的ruby实现设置为默认值，这样'哪个ruby'显示'~/.rvm/ruby-1.8.6-p383/bin/ruby'但是当我在emacs中打开inf-ruby缓冲区时，它使用安装在/usr/bin中的ruby。有没有办法让emacs像shell一样尊重ruby的路径？谢谢! 最佳答案 我创建了一个emacs扩展来将rvm集成到emacs中。如果您有兴趣，可以在这里获取:http://github.com/senny/rvm.el
假设我有这个范围:("aaaaa".."zzzzz")如何在不事先/每次生成整个项目的情况下从范围中获取第N个项目？ 最佳答案 一种快速简便的方法:("aaaaa".."zzzzz").first(42).last#==>"aaabp"如果出于某种原因你不得不一遍又一遍地这样做，或者如果你需要避免为前N个元素构建中间数组，你可以这样写:moduleEnumerabledefskip(n)returnto_enum:skip,nunlessblock_given?each_with_indexdo|item,index|yieldit
我目前正在使用以下方法获取页面的源代码:Net::HTTP.get(URI.parse(page.url))我还想获取HTTP状态，而无需发出第二个请求。有没有办法用另一种方法做到这一点？我一直在查看文档，但似乎找不到我要找的东西。 最佳答案 在我看来，除非您需要一些真正的低级访问或控制，否则最好使用Ruby的内置Open::URI模块:require'open-uri'io=open('http://www.example.org/')#=>#body=io.read[0,50]#=>"["200","OK"]io.base_ur
如何在Ruby中获取BasicObject实例的类名？例如，假设我有这个:classMyObjectSystem我怎样才能使这段代码成功？编辑:我发现Object的实例方法class被定义为returnrb_class_real(CLASS_OF(obj));。有什么方法可以从Ruby中使用它？ 最佳答案 我花了一些时间研究irb并想出了这个:classBasicObjectdefclassklass=class这将为任何从BasicObject继承的对象提供一个#class您可以调用的方法。编辑评论中要求的进一步解释:假设你有对象
是否可以在应用程序中包含的gem代码中知道应用程序的Rails文件系统根目录？这是gem来源的示例:moduleMyGemdefself.included(base)putsRails.root#returnnilendendActionController::Base.send:include,MyGem谢谢，抱歉我的英语不好 最佳答案 我发现解决类似问题的解决方案是使用railtie初始化程序包含我的模块。所以，在你的/lib/mygem/railtie.rbmoduleMyGemclassRailtie使用此代码，您的模块将在