Hadoop runs in three forms:

• Single instance run
• Pseudo distributed
• Fully distributed
  This article focuses on the installation and use of single-instance Hadoop. The following installation and configuration steps are primarily for quick installation and experiencing Hadoop, rather than using Hadoop in a production environment.

Single instance Hadoop

Download

FromHadoop download homepageChoose the appropriate version to download Hadoop.
An example of the command is as follows:

HADOOP_VERSION=hadoop-3.0.0-alpha1
wget http://apache.fayea.com/hadoop/common/${HADOOP_VERSION}/${HADOOP_VERSION}.tar.gz
wget https://dist.apache.org/repos/dist/release/hadoop/common/${HADOOP_VERSION}/${HADOOP_VERSION}.tar.gz.mds
wget https://dist.apache.org/repos/dist/release/hadoop/common/${HADOOP_VERSION}/${HADOOP_VERSION}.tar.gz.asc
wget https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

2. Verification and decompression

• verification
  The command performs similarly as follows:

  HADOOP_VERSION=hadoop-3.0.0-alpha1
  md5 ${HADOOP_VERSION}.tar.gz | awk '{print $4}' > mycomputation.md5
  sed -n '1,2p' ${HADOOP_VERSION}.tar.gz.mds | sed 's/[ ]//g' | awk -v RS="" -F= '{gsub("\n","");print tolower($2)}' > extraction.md5
  diff mycomputation.md5 extraction.md5
  gpg -q --import KEYS
  gpg --verify ${HADOOP_VERSION}.tar.gz.asc ${HADOOP_VERSION}.tar.gz
  

• Decompression
  Verify that the downloaded file is extracted and goes to its root directory.
  The command performs similarly as follows:

  HADOOP_VERSION=hadoop-3.0.0-alpha1
  tar xzvf ${HADOOP_VERSION}.tar.gz
  cd ${HADOOP_VERSION}
  

3. Set the JAVA_HOME environment variable

Change settingJAVA_HOME。
note:
is by modifying the fileetc/hadoop/hadoop-env.shmiddleJAVA_HOMEAttribute variable.
The example is as follows:

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home/

4. Test installation is normal

• By executionbin/hadoop
  execute the commandbin/hadoopNormally, you should display the hadoop command usage documentation.

• Run the map reduce task
  Execute the following command:

  mkdir input
  cp etc/hadoop/*.xml input
  bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-alpha1.jar grep input output 'dfs[a-z.]+'
  cat output/*
  

Pseudo-distributed Hadoop

1. Configuration

• Edit fileetc/hadoop/core-site.xml:

  <configuration>
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://localhost:9000</value>
      </property>
  </configuration>
  

• Edit fileetc/hadoop/hdfs-site.xml:

  <configuration>
      <property>
          <name>dfs.replication</name>
          <value>1</value>
      </property>
  </configuration>
  

2. No password ssh login

Execute the following command:

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

3. Execute the test

1. Format the file system:
   bin/hdfs namenode -format

2. Start the NameNode daemon and the DataNode daemon:

   sbin/start-dfs.sh
   

Write the log$HADOOP_LOG_DIRDirectory (default is$HADOOP_HOME/logs)。

3. Browse the web interface of the NameNode:

• NameNode - http://localhost:9870/

4. Create an HDFS directory to perform MapReduce tasks:

   bin/hdfs dfs -mkdir /user
   bin/hdfs dfs -mkdir /user/zhengfq
   

5. Copy the input file to the distributed file system:

   bin/hdfs dfs -mkdir input
   bin/hdfs dfs -put etc/hadoop/*.xml input
   

6. Running example:

    bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-alpha1.jar grep input output 'dfs[a-z.]+'
   

7. Check the output:

    bin/hdfs dfs -cat output/*
   

8. Stop the process:

    sbin/stop-dfs.sh
   

reference

• Apache Hadoop homepage
• Apache Hadoop documentation
• Build and Install Hadoop 2.x or newer on Windows
• Single-node Hadoop
• Hadoop YARN Installation: The definitive guide