到目前为止，对于这个问题，我已经尝试了这里的解决方案，1 ，在这里，2 .然而，虽然这些解决方案确实导致执行 mapreduce 任务，但看起来它们只在名称节点上运行，因为我得到类似于此处的输出，3。 .

基本上，我正在使用我自己设计的 mapreduce 算法运行一个2 节点集群。 mapreduce jar 在单节点集群上完美执行，这让我觉得我的 hadoop 多节点配置有问题。要设置多节点，我遵循了教程 here .

为了报告出了什么问题，当我执行我的程序时(在检查名称节点、任务跟踪器、作业跟踪器和数据节点正在各自的节点上运行之后)，我的程序在终端中的这一行停止:

INFO mapred.JobClient:map 100% reduce 0%

如果我查看任务日志，我看到copy failed: attempt... from slave-node 后跟SocketTimeoutException.

查看我的从节点 (DataNode) 上的日志显示执行在以下行停止:

TaskTracker:尝试...减少 0.0% > 复制 >

正如链接 1 和 2 中的解决方案所建议的那样，从 etc/hosts 文件中删除各种 ip 地址会导致成功执行，但是我最终得到了诸如在我的slave-node (DataNode) log 的链接 4 中，例如:

信息 org.apache.hadoop.mapred.TaskTracker:收到“KillJobAction” 工作:job_201201301055_0381

警告 org.apache.hadoop.mapred.TaskTracker:未知作业 job_201201301055_0381 被删除。

作为一个新的 hadoop 用户，这看起来对我来说很可疑，但看到这个可能是完全正常的。对我来说，这看起来好像有什么东西指向主机文件中不正确的 ip 地址，并且通过删除这个 ip 地址，我只是停止在从节点上执行，并且处理改为在名称节点上继续(这根本没有什么好处)。

总结:

1. 这是预期的输出吗？
2. 有什么方法可以查看执行后在哪个节点上执行了什么？
3. 有人能发现我做错了什么吗？

编辑为每个节点添加主机和配置文件

主人:etc/hosts

127.0.0.1       localhost
127.0.1.1       joseph-Dell-System-XPS-L702X

#The following lines are for hadoop master/slave setup
192.168.1.87    master
192.168.1.74    slave

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

从机:etc/hosts

127.0.0.1       localhost
127.0.1.1       joseph-Home # this line was incorrect, it was set as 7.0.1.1

#the following lines are for hadoop mutli-node cluster setup
192.168.1.87    master
192.168.1.74    slave

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

掌握:core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hduser/tmp</value>
    <description>A base for other temporary directories.</description>
</property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://master:54310</value>
        <description>The name of the default file system. A URI whose
        scheme and authority determine the FileSystem implementation. The
        uri’s scheme determines the config property (fs.SCHEME.impl) naming
        the FileSystem implementation class. The uri’s authority is used to
        determine the host, port, etc. for a filesystem.</description>
    </property>
</configuration>

从站:core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hduser/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>

    <property>
        <name>fs.default.name</name>
        <value>hdfs://master:54310</value>
        <description>The name of the default file system. A URI whose
        scheme and authority determine the FileSystem implementation. The
        uri’s scheme determines the config property (fs.SCHEME.impl) naming
        the FileSystem implementation class. The uri’s authority is used to
        determine the host, port, etc. for a filesystem.</description>
    </property>

</configuration>

母版:hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
        <description>Default block replication.
        The actual number of replications can be specified when the file is created.
        The default is used if replication is not specified in create time.
        </description>
    </property>
</configuration>

从站:hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
        <description>Default block replication.
        The actual number of replications can be specified when the file is created.
        The default is used if replication is not specified in create time.
        </description>
    </property>
</configuration>

掌握:mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
    <property>
        <name>mapred.job.tracker</name>
        <value>master:54311</value>
        <description>The host and port that the MapReduce job tracker runs
        at. If “local”, then jobs are run in-process as a single map
        and reduce task.
        </description>
    </property>
</configuration>

从站:mapre-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>
        <name>mapred.job.tracker</name>
        <value>master:54311</value>
        <description>The host and port that the MapReduce job tracker runs
        at. If “local”, then jobs are run in-process as a single map
        and reduce task.
        </description>
    </property>

</configuration>

最佳答案

错误在 etc/hosts:

在错误运行期间，slave etc/hosts 文件如下所示:

127.0.0.1       localhost
7.0.1.1       joseph-Home # THIS LINE IS INCORRECT, IT SHOULD BE 127.0.1.1

#the following lines are for hadoop mutli-node cluster setup
192.168.1.87    master
192.168.1.74    slave

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

您可能已经发现，这台计算机“joseph-Home”的 IP 地址配置不正确。当它应该设置为 127.0.1.1 时，它被设置为 7.0.1.1。因此，将 slave etc/hosts 文件的第 2 行更改为 127.0.1.1 joseph-Home 解决了这个问题，并且我的日志在从属节点上正常显示。

新的 etc/hosts 文件:

127.0.0.1       localhost
127.0.1.1       joseph-Home # THIS LINE IS INCORRECT, IT SHOULD BE 127.0.1.1

#the following lines are for hadoop mutli-node cluster setup
192.168.1.87    master
192.168.1.74    slave

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

关于java - Hadoop 集群卡住卡在 Reduce > copy >，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/18634825/