下载软件hadoop3.1版本

linux 操作系统 centos镜像下载 http://mirrors.aliyun.com/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso

linux jdk https://download.oracle.com/otn-pub/java/jdk/8u191-b12/2787e4a523244c269598db4e85c51e0c/jdk-8u191-linux-x64.tar.gz?AuthParam=1547309437_68428895cdc4d2e411123cd02d6750e4

hadoop下载 http://archive.apache.org/dist/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz

准备3台机器

用户权限

brroot 超级管理员（系统文件修改）brbrhd 普通角色用户 （在/home/hd/*   进行所有增删改查）brbr密码统一设置成：hd 或者 123456

修改静态IP地址

第一台机器

设置网卡为静态的地址

br切换root用户br[hd@bogon Desktop]$ su rootbrPassword: br修改网卡br[root@bogon Desktop]# vim /etc/sysconfig/network-scripts/ifcfg-eth0 brDEVICE="eth0"brBOOTPROTO="static"     #brONBOOT="yes"brTYPE="Ethernet"brIPADDR=192.168.245.20  #[2-254]br重启网卡br[root@bogon Desktop]# service network restart br查看IP信息br[root@bogon Desktop]# ifconfig 

第二台机器

br切换root用户br[hd@bogon Desktop]$ su rootbrPassword: br修改网卡br[root@bogon Desktop]# vim /etc/sysconfig/network-scripts/ifcfg-eth0 brDEVICE="eth0"brBOOTPROTO="static"     #brONBOOT="yes"brTYPE="Ethernet"brIPADDR=192.168.245.30br删除电脑网卡与系统绑定关系br[root@localhost Desktop]# rm -rf /etc/udev/rules.d/70-persistent-net.rules br重启系统br[root@localhost Desktop]# rebootbr查看IP信息br[root@bogon Desktop]# ifconfig 

第三台机器

br切换root用户br[hd@bogon Desktop]$ su rootbrPassword: br修改网卡br[root@bogon Desktop]# vim /etc/sysconfig/network-scripts/ifcfg-eth0 brDEVICE="eth0"brBOOTPROTO="static"     #brONBOOT="yes"brTYPE="Ethernet"brIPADDR=192.168.245.40br删除电脑网卡与系统绑定关系br[root@localhost Desktop]# rm -rf /etc/udev/rules.d/70-persistent-net.rules br重启系统br[root@localhost Desktop]# rebootbr查看IP信息br[root@bogon Desktop]# ifconfigbrbr

获取机器的IP地址

br[root@localhost ~]# ifconfig breno16777736: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500br        inet 192.168.20.46  netmask 255.255.255.0  broadcast 192.168.20.46brbr#如果没有查看IP的命令就安装步骤br[hd@localhost root]$ su rootbrPassword: br[root@localhost ~]# yum install -y net-toolsbrbr#也可查看ip addr 命令查看IP地址。 br[hd@slave02 ~]$ ip addrbr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000br    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00br    inet 127.0.0.1/8 scope host lobr       valid_lft forever preferred_lft foreverbr    inet6 ::1/128 scope host br       valid_lft forever preferred_lft foreverbr2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000br    link/ether 00:0c:29:4f:71:df brd ff:ff:ff:ff:ff:ffbr    inet 192.168.20.46/24 brd 192.168.20.255 scope global noprefixroute dynamic ens160br       valid_lft 32521sec preferred_lft 32521secbr    inet6 fe80::f52e:651e:721b:a15a/64 scope link tentative noprefixroute dadfailed br       valid_lft forever preferred_lft foreverbr    inet6 fe80::5ebb:a198:b90f:fe68/64 scope link tentative noprefixroute dadfailed br       valid_lft forever preferred_lft foreverbr    inet6 fe80::c180:4b3e:a100:224c/64 scope link tentative noprefixroute dadfailed br       valid_lft forever preferred_lft foreverbr

Linux安装Java环境

Linux常用的命令

brsu 切换用户brbrpwd 显示当前目录brbrtar 解压，压宿brbrcd 进入目录brbrmv 移动文件，也可以改文件名brbrsource /etc/profile 加载环境变量brbrvi 文本编辑器，相当于我们的记事本一样，但是使用的都是命令的操作。

把Linux自带Java环境删除

br[hd@localhost ~]$ su root brPassword: br[root@localhost hd]# yum remove  -y  java*  

上传Java包

br[root@localhost hd]# su hdbr[hd@localhost ~]$ br[hd@localhost ~]$ br[hd@localhost ~]$ pwdbr/home/hdbr[hd@localhost ~]$ mkdir apps #上传到此目录 br[hd@localhost ~]$ cd apps/br[hd@localhost apps]$ br#上传过程br[hd@localhost apps]$ llbrtotal 178952br-rw-rw-r--. 1 hd hd 183246769 Apr 26  2018 jdk-8u121-linux-x64.tar.gz

解压java包

br#解压br[hd@localhost apps]$ tar -zxvf jdk-8u121-linux-x64.tar.gz br[hd@localhost apps]$ llbrtotal 178956brdrwxr-xr-x. 8 hd hd      4096 Dec 12  2016 jdk1.8.0_121br-rw-rw-r--. 1 hd hd 183246769 Apr 26  2018 jdk-8u121-linux-x64.tar.gzbr[hd@localhost apps]$ br#目录改名br[hd@localhost apps]$ mv jdk1.8.0_121/ java  br[hd@localhost apps]$ llbrtotal 178956brdrwxr-xr-x. 8 hd hd      4096 Dec 12  2016 javabr-rw-rw-r--. 1 hd hd 183246769 Apr 26  2018 jdk-8u121-linux-x64.tar.gz

配置java环境

br[hd@localhost apps]$ su rootbrPassword: br[root@localhost apps]# cd java/br[root@localhost java]# pwdbr/home/hd/apps/javabr[root@localhost java]# br[root@localhost java]# vi /etc/profile

使用vi编辑器，在/etc/profile增加java环境变量

brexport JAVA_HOME=/home/hd/apps/javabrexport PATH=$PATH:$JAVA_HOME/bin

重加载一下系统环境

br[root@localhost java]# source /etc/profilebr[root@localhost java]# java -versionbrjava version "1.8.0_121"brJava(TM) SE Runtime Environment (build 1.8.0_121-b13)brJava HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

配置第二台，第三台机器的Java环境

使用scp远程拷贝命令

brscp  file2  [[user@]host2:]file2

1.把每一台机的java目录，拷贝到第二台机器

br[root@localhost apps]# su hdbr[hd@localhost apps]$ br[hd@localhost apps]$  scp -r java  hd@192.168.126.129:/home/hd/apps/

2.把每一台机的profile文件，拷贝到第二台机器

br[hd@localhost apps]$ su rootbrPassword: br[root@localhost apps]# scp /etc/profile root@192.168.126.129:/etc/brThe authenticity of host '192.168.126.129 (192.168.126.129)' can't be established.brECDSA key fingerprint is fb:0a:7a:9f:9a:bc:4f:ff:66:29:1d:1d:b9:a0:35:d1.brAre you sure you want to continue connecting (yes/no)? yesbrWarning: Permanently added '192.168.126.129' (ECDSA) to the list of known hosts.brroot@192.168.126.129's password: brprofile                                                                                                        100% 1820     1.8KB/s   00:00    br[root@localhost apps]# 

3.第二台机器加载profile

br[hd@localhost apps]$ source /etc/profilebr[hd@localhost apps]$ br[hd@localhost apps]$ java -versionbrjava version "1.8.0_121"brJava(TM) SE Runtime Environment (build 1.8.0_121-b13)brJava HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)br[hd@localhost apps]$ 

第三台机，执行以上的步骤

安装hadoop之前准备

1.修改主机名

1. 第一台机器master
2. 第二台机器slave01
3. 第三台机器slave02

br[hd@localhost ~]$ hostnamectl set-hostname masterbr==== AUTHENTICATING FOR org.freedesktop.hostname1.set-static-hostname ===brAuthentication is required to set the statically configured local host name, as well as the pretty host name.brAuthenticating as: rootbrPassword: br==== AUTHENTICATION COMPLETE ===

br[hd@localhost ~]$ hostnamectl set-hostname slave01

br[hd@localhost ~]$ hostnamectl set-hostname slave02

2.修改/etc/hosts 文件

br[hd@master ~]$ su rootbrPassword: br[root@master hd]# vi /etc/hostsbr127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4br::1         localhost localhost.localdomain localhost6 localhost6.localdomain6brbr192.168.126.128 masterbr192.168.126.129 slave01br192.168.126.130 slave02

同步到第二，三台机器

br#第二台机器br[root@master hd]# scp /etc/hosts root@slave01:/etc/br#第三台机器br[root@master hd]# scp /etc/hosts root@slave02:/etc/

3.关闭防火墙

br启动：systemctl start firewalld.servicebrbr查看状态： systemctl status firewalld.servicebrbr停止： systemctl stop firewalld.servicebrbr禁用： systemctl disable firewalld.service brbr重启: systemctl  restart firewalld.service

4.免密登录

需要做的免密的机器

生成密钥

br[hd@master ~]$ ssh-keygen brGenerating public/private rsa key pair.brEnter file in which to save the key (/home/hd/.ssh/id_rsa): brEnter passphrase (empty for no passphrase): brEnter same passphrase again: brYour identification has been saved in /home/hd/.ssh/id_rsa.brYour public key has been saved in /home/hd/.ssh/id_rsa.pub.brThe key fingerprint is:bref:ff:98:6c:a4:66:ca:66:a0:cd:a4:da:75:9c:c0:9f hd@slave02brThe key's randomart image is:br+--[ RSA 2048]----+br|                 |br|                 |br|                 |br|     .           |br|      o S        |br|      o+ +  .    |br|     *..E .o     |br|   .o.ooo.+..o   |br|  ...  oo+.o=..  |br+-----------------+

拷贝密钥到你需要免密登录的机器

br[hd@master ~]$ ssh-copy-id slave02brThe authenticity of host 'slave02 (192.168.126.130)' can't be established.brECDSA key fingerprint is 09:57:a3:56:3b:5f:f0:01:55:0e:42:f3:4c:43:3d:d5.brAre you sure you want to continue connecting (yes/no)? yesbr/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installedbr/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keysbrhd@slave02's password: brbrNumber of key(s) added: 1brbrNow try logging into the machine, with:   "ssh 'slave02'"brand check to make sure that only the key(s) you wanted were added.

测试免密登录

br[hd@master ~]$ ssh slave01brLast login: Sun Jul 12 23:54:32 2020 from 192.168.22.130br[hd@slave01 ~]$ br[hd@slave01 ~]$ 

Hadoop安装

上传hadoop安装包

br[hd@master hadoop]$ rzbrrz waiting to receive.br zmodem trl+C ?

解压安装包

br[hd@master apps]$ su hdbrPassword: br[hd@master apps]$ pwdbr/home/hd/appsbr[hd@master apps]$ tar -zxvf hadoop-3.0.0.tar.gz 

改目录名称

brhd@master apps]$ mv hadoop-3.0.0 hadoopbr[hd@master apps]$ llbrtotal 324644brdrwxr-xr-x. 12 hd hd       192 Jul 11 00:09 hadoop

修改hadoop配置文件

修改hadoop-env.sh

br[hd@master ~]$ cd /home/hd/apps/hadoop/etc/hadoop/br[hd@master hadoop]$ br[hd@master hadoop]$ pwdbr/home/hd/apps/hadoop/etc/hadoopbr[hd@master hadoop]$ vi hadoop-env.shbrbr#在文件的尾部（按“G”可以跳到文档的尾部），增加brexport JAVA_HOME=/home/hd/apps/java

修改core-site.xml

br<configuration>br    <!-- 指定HADOOP所使用的文件系统schema（URI），HDFS的老大（NameNode）的地址 -->br    <property>br        <name>fs.defaultFS</name>br        <value>hdfs://master:9000</value>br    </property>br    <!-- 指定hadoop运行时产生文件的存储目录 -->br    <property>br        <name>hadoop.tmp.dir</name>br        <value>/home/hd/apps/hadoop/tmpdata</value>br    </property>br</configuration>

修改hdfs-site.xml

br<configuration>br    <!-- 指定HDFS副本的数量 -->br    <property>br        <name>dfs.replication</name>br        <value>2</value>br    </property>br    <!-- 设置namenode的http通讯地址 -->br    <property>br        <name>dfs.namenode.http-address</name>br        <value>master:50070</value>br    </property>brbr    <!-- 设置secondarynamenode的http通讯地址 -->br    <property>br        <name>dfs.namenode.secondary.http-address</name>br        <value>master:50090</value>br    </property>brbr    <!-- 设置namenode存放的路径 -->br    <property>br        <name>dfs.namenode.name.dir</name>br        <value>/home/hd/apps/hadoop/namenode</value>br    </property>brbr    <!-- 设置datanode存放的路径 -->br    <property>br        <name>dfs.datanode.data.dir</name>br        <value>/home/hd/apps/hadoop/datanode</value>br    </property>br</configuration>

修改mapred-site.xml

br<configuration> br    <!-- 指定mr运行在yarn上 -->br    <property>br        <name>mapreduce.framework.name</name>br        <value>yarn</value>br    </property>br    <property>br        <name>yarn.app.mapreduce.am.env</name>br        <value>HADOOP_MAPRED_HOME=/home/hd/apps/hadoop</value>br    </property>br    <property>br          <name>mapreduce.map.env</name>br          <value>HADOOP_MAPRED_HOME=/home/hd/apps/hadoop</value>br    </property>br    <property>br          <name>mapreduce.reduce.env</name>br          <value>HADOOP_MAPRED_HOME=/home/hd/apps/hadoop</value>br    </property>br</configuration>

修改yarn-site.xml

br<configuration>br    <!-- 指定YARN的老大（ResourceManager）的地址 -->br    <property>br        <name>yarn.resourcemanager.hostname</name>br        <value>master</value>br    </property>br    <!-- reducer获取数据的方式 -->br    <property>br        <name>yarn.nodemanager.aux-services</name>br        <value>mapreduce_shuffle</value>br    </property>br</configuration>

修改workers

br[hd@master hadoop]$ vi workers brslave01brslave02

修改环境变量

br[hd@master hadoop]$ su root brPassword: br[root@master hadoop]# vi /etc/profilebrbr#增加brexport HADOOP_HOME=/home/hd/apps/hadoopbr#增加brexport PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

拷贝到第二，三台机

br[root@master hadoop]# su hdbr[hd@master hadoop]$ br[hd@master hadoop]$ br[hd@master hadoop]$ scp  -r /home/hd/apps/hadoop hd@slave01:/home/hd/apps/br[hd@master hadoop]$ scp  -r /home/hd/apps/hadoop hd@slave02:/home/hd/apps/

br[hd@master hadoop]$ su rootbrPassword: br[root@master hadoop]# scp /etc/profile root@slave01:/etc/brroot@slave01's password: brprofile                                                                                                        100% 1896     1.9KB/s   00:00    br[root@master hadoop]# scp /etc/profile root@slave02:/etc/brprofile                                                                                                        100% 1896     1.9KB/s   00:00    br[root@master hadoop]# 

在第三台机器里加载环境

br[root@master hadoop]#  source /etc/profilebrbr[hd@master hadoop]$ hadoop versionbrHadoop 3.0.0

格式化

br[hd@master hadoop]$ ll /home/hd/apps/hadoop/namenodebrls: cannot access /home/hd/apps/hadoop/namenode: No such file or directorybr[hd@master hadoop]$ hadoop namenode -format 

启动hadoop

start-dfs.sh 启动HDFS分布式文件系统，停止stop-dfs.sh

start-yarn.sh 启动Yarn资源管理器，停止stop-yarn.sh

start-all.sh HDFS分布式文件系统与Yarn启动，停止stop-all.sh

jps查看进程

br[hd@master ~]$ jpsbr23668 SecondaryNameNodebr23467 NameNodebr23903 ResourceManagerbr24207 Jps

br[hd@slave01 ~]$ jpsbr22341 DataNodebr22649 Jpsbr22458 NodeManager

br[hd@slave02 ~]$ jpsbr23367 Jpsbr23176 NodeManagerbr23051 DataNode

测试

hdfs 文件系统访问地址：http://192.168.126.128:50070/dfshealth.html#tab-overview

Yarn资源管理器访问地址：http://192.168.126.128:8088/cluster