tags: Hadoop Cluster installation
1, must bind the host name and IP, otherwise the remote will not connect
vi /etc/hosts
10.0.6.29 commonhost
2, set ssh free login
(1) ssh-keygen, always enter
(2)ssh-copy-id localhost
3, install Java
(1) Install jdk1.8:
yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel
(2) Set environment variables:
vi /etc/profile
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre
export CLASSPATH=.:$JAVA_HOME/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
source /etc/profile
(3) View the installation results:
echo $JAVA_HOME
echo $CLASSPATH
echo $PATH
java -version
4、vi /etc/profile
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_HOME=/home/hadoop
Source /etc/profile makes it effective
5. Upload and unzip hadoop-3.0.3.tar.gz
6, modify the configuration file under hadoop/etc/hadoop
(1)hadoop-env.sh:
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre
(2)core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://commonhost:9000/</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmp/</value>
</property>
</configuration>
(3)hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
(4)mapred-site.xml:
<configuration>
<property>
<name>maperduce.framwork.name</name>
<value>yarn</value>
</property>
</configuration>
(5)yarn-site.xml:
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>commonhost</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>maperduce_shuffle</value>
</property>
</configuration>
If you need to deploy a cluster, execute 7-10
7. Set ssh to login on the other two machines and install Java.
8, configure the master/slave under etc/hadoop
vi workers
10.0.6.29
10.0.6.39
10.0.6.49
9, remotely copy hadoop to two machines
scp -r /home/hadoop 10.0.6.39:/home/hadoop
scp -r /home/hadoop 10.0.6.49:/home/hadoop
10, remotely copy /etc/profile to two machines
scp -r /etc/profile 10.0.6.39:/etc/profile
scp -r /etc/profile 10.0.6.49:/etc/profile
11, each server bin. /hadoop namenode -format
12, start the master in the sbin directory, single machine and cluster only need to start one
./start-all.sh
jps view
13, connection test
http://10.0.6.29:9870 //HDFS
14, MapReduce test
cd /home/hadoop/share/hadoop
hadoop jar hadoop-mapreduce-examples-3.0.3.jar pi 5 5
hadoop jar hadoop-mapreduce-examples-3.0.3.jar wordcount /work/test.txt /output
At the beginning of the year, I participated in the overall stack of Huawei, including Java, MySQL, Hadoop, Spark, is considered a big data smaller update cognitive system. Planned address: Today, I w...
**precondition: VMware installed successfully Three CentOS installations success ** continue: The first step: modify the system name, and the map of the IP and name of the three systems. Vim / etc / h...
Reference blog: Hadoop pseudo-distribution Hadoop's environment in linux Hadoop installation to build a pseudo -distributed tutorial (comprehensive) vomiting blood compilation Before You need to confi...
The first section of the Hadoop infrastructure configuration 1.0 pre-action: Make the ordinary user I created get administrator rights is as follows 1) Switch to the administrator user, the switching ...
1. Install Hadoop 2.7 in CentOS 1.1 Setting Static IP 1.2 Modifying the host name 1.3 Bind hostname to ip 1.4 Turn off the firewall 1.5 Turn off the automatic operation of the firewall 1.6 SSH (Secure...
Hadoop is the distributed computing platform of the current comparative process. Although the official website of the installation and usage method is more detailed, the details are quite a lot...
Hadoop runs in three forms: Single instance run Pseudo distributed Fully distributed This article focuses on the installation and use of single-instance Hadoop. The following installation and configur...
1. the widget named hadoop-eclipse-plugin-2.7.5.jar in ECLIPSE_HOME / plugins directory 2. hadoop-2.7.5-centos-6.7.tar.gz extract to a local computer 3. Configuration windows environment variab...
(1) Which Linux distribution to choose 1. The CentOS system and Ubuntu system are the most used on the server and desktop side of each distribution of the Linux system, and the information on the netw...
Preparation before installation Check if java is installed, hadoop requires java commandjava -version This is installed Configure your own environment and executessh localhost If you don't have permis...