Hadoop notes (a) installation and use

tags: Hadoop  Cluster  installation

1, must bind the host name and IP, otherwise the remote will not connect
vi /etc/hosts

10.0.6.29  commonhost

2, set ssh free login
(1) ssh-keygen, always enter
(2)ssh-copy-id localhost

3, install Java
(1) Install jdk1.8:

yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel

(2) Set environment variables:
vi /etc/profile

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre
export CLASSPATH=.:$JAVA_HOME/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
source /etc/profile

(3) View the installation results:
echo $JAVA_HOME
echo $CLASSPATH
echo $PATH
java -version

4、vi /etc/profile

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_HOME=/home/hadoop

Source /etc/profile makes it effective

5. Upload and unzip hadoop-3.0.3.tar.gz

6, modify the configuration file under hadoop/etc/hadoop
(1)hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre

(2)core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://commonhost:9000/</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hadoop/tmp/</value>
    </property>
</configuration>

(3)hdfs-site.xml:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
	</property>
</configuration>

(4)mapred-site.xml:

<configuration>
    <property>
        <name>maperduce.framwork.name</name>
        <value>yarn</value>
	</property>
</configuration>

(5)yarn-site.xml:

<configuration>
     <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>commonhost</value>
	</property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>maperduce_shuffle</value>
    </property>
</configuration>

If you need to deploy a cluster, execute 7-10
7. Set ssh to login on the other two machines and install Java.

8, configure the master/slave under etc/hadoop
vi workers

10.0.6.29
10.0.6.39
10.0.6.49

9, remotely copy hadoop to two machines
scp -r /home/hadoop 10.0.6.39:/home/hadoop
scp -r /home/hadoop 10.0.6.49:/home/hadoop

10, remotely copy /etc/profile to two machines
scp -r /etc/profile 10.0.6.39:/etc/profile
scp -r /etc/profile 10.0.6.49:/etc/profile

11, each server bin. /hadoop namenode -format

12, start the master in the sbin directory, single machine and cluster only need to start one
./start-all.sh
jps view

13, connection test
http://10.0.6.29:9870 //HDFS

14, MapReduce test
cd /home/hadoop/share/hadoop
hadoop jar hadoop-mapreduce-examples-3.0.3.jar pi 5 5
hadoop jar hadoop-mapreduce-examples-3.0.3.jar wordcount /work/test.txt /output

Intelligent Recommendation

Hadoop 3.3.1 Installation Notes

At the beginning of the year, I participated in the overall stack of Huawei, including Java, MySQL, Hadoop, Spark, is considered a big data smaller update cognitive system. Planned address: Today, I w...

Hadoop installation notes (2)

**precondition: VMware installed successfully Three CentOS installations success ** continue: The first step: modify the system name, and the map of the IP and name of the three systems. Vim / etc / h...

Hadoop installation configuration notes

Reference blog: Hadoop pseudo-distribution Hadoop's environment in linux Hadoop installation to build a pseudo -distributed tutorial (comprehensive) vomiting blood compilation Before You need to confi...

Hadoop installation and use

The first section of the Hadoop infrastructure configuration 1.0 pre-action: Make the ordinary user I created get administrator rights is as follows 1) Switch to the administrator user, the switching ...

Hadoop installation and initial use

1. Install Hadoop 2.7 in CentOS 1.1 Setting Static IP 1.2 Modifying the host name 1.3 Bind hostname to ip 1.4 Turn off the firewall 1.5 Turn off the automatic operation of the firewall 1.6 SSH (Secure...

More Recommendation

[ ]hadoop installation and use

  Hadoop is the distributed computing platform of the current comparative process. Although the official website of the installation and usage method is more detailed, the details are quite a lot...

Introduction to Hadoop installation and use

Hadoop runs in three forms: Single instance run Pseudo distributed Fully distributed This article focuses on the installation and use of single-instance Hadoop. The following installation and configur...

Use the eclipse hadoop installation

1. the widget named hadoop-eclipse-plugin-2.7.5.jar in ECLIPSE_HOME / plugins directory 2. hadoop-2.7.5-centos-6.7.tar.gz extract to a local computer   3. Configuration windows environment variab...

Installation and use of Hadoop

(1) Which Linux distribution to choose 1. The CentOS system and Ubuntu system are the most used on the server and desktop side of each distribution of the Linux system, and the information on the netw...

Mac Hadoop installation and use

Preparation before installation Check if java is installed, hadoop requires java commandjava -version This is installed Configure your own environment and executessh localhost If you don't have permis...

Copyright  DMCA © 2018-2026 - All Rights Reserved - www.programmersought.com  User Notice

Top