master 192.168.132.131
slave3 192.168.132.141
slave2 192.168.132.144
slave3 192.168.132.142
master和node节点都要添加到/etc/hosts
192.168.132.144 slave2192.168.132.131 master192.168.132.142 slave1192.168.132.141 slave3
一、配置ssh无秘钥登录
ssh-keygen -t rsa -P ''#关闭防火墙:(centos7)systemctl stop firewalld.service #停止firewallsystemctl disable firewalld.service #禁止firewall开机启动
二、java1.8 安装
环境变量设置
cat /etc/proifileexport HADOOP_HOME=/usr/local/hadoop (hadoop 环境变量)export PATH=$PATH:$HADOOP_HOME/binexport JAVA_HOME=/usr/local/java/jdk1.8.0_131/export JRE_HOME=/usr/local/java/jdk1.8.0_131/jreexport CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/libexport PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
三 、安装hadoop 2.8.1
解压,放到/usr/local/hadoop目录,配置文件目录
cd /usr/local/hadoop/etc/hadoopmkdir /usr/local/hadoop/{hdfs,tmp}mkdir /usr/local/hadoop/hdfs/{name,tmp,data}
四、配置文件(master服务器)
core-site.xml
hadoop.tmp.dir /usr/local/hadoop/tmp true A base for other temporary directories. fs.default.name hdfs://192.168.132.131:9000 ##master iptrue io.file.buffer.size 131072
配置hdfs-site.xml文件
dfs.replication 2 #数据备份的个数dfs.name.dir /usr/local/hadoop/hdfs/name #namenode持久存储名字的额空间、事物日志路径dfs.data.dir /usr/local/hadoop/hdfs/data #datanode数据存储路径dfs.namenode.secondary.http-address 192.168.132.131:9001 #master ipdfs.webhdfs.enabled true dfs.permissions false
配置mapred-site.xml文件
mapred.job.tracker http://192.168.132.131:9001 #master ipmapreduce.framework.name yarn
配置yarn-site.xml文件
yarn.resourcemanager.address 192.168.132.131:18040 yarn.resourcemanager.scheduler.address 192.168.132.131:18030 yarn.resourcemanager.webapp.address 192.168.132.131:18088 yarn.resourcemanager.resource-tracker.address 192.168.132.131:18025 yarn.resourcemanager.admin.address 192.168.132.131:18141 yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler
配置master文件
配置slaves文件(master节点独有)
五、配置所有slave节点
将master节点的hadoop 以上配置的文件拷贝过去即可(不包括slaves文件)
六、启动服务(master节点)
hadoop namenode –format
启动 /关闭 hadoop ,/usr/local/hadoop/sbin 目录下
start-all.sh //此命令会启动集群所有节点
stop-all.sh //此命令会启动集群所有节点
启动单个节点命令,单个namenode,datanode,ResourceManager ,nodemanager的命令
hadoop namenode -format hadoop-daemon.sh start namenode hadoop-daemons.sh start datanode yarn-daemon.sh start resourcemanager yarn-daemons.sh start nodemanager
测试:
master节点
slave节点
查看集群状态: hadoop dfsadmin -report
通过网页查看集群 masterip:50070
http://192.168.132.131:18088/cluster
MapReduce测试
[root@master mapreduce]# hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar pi 10 100