1. 脚本编写注意事项
1.1. 使用 linux vim 编辑器
- 脚本编写，最好先在 Windows 环境下，编辑好，再粘贴到 linux 的 vim 编辑器；
- 不能直接将脚本推送到 linux 中，否则执行的时候会因为回车问题，导致脚本无法正常运行。
1.2. 脚本存放到 bin 目录下
- 写好的脚本放到 /bin 目录下，可以直接执行；
1.3. 修改 777 权限
chmod 777 脚本名称
1.4. 注意脚本后缀
- 不要后缀的脚本要注意；
2. xsync
2.1. xsync 简介
改脚本基于 rsync 而写的，rsync 与 scp 的运行机制不一样，rsync 会对比集群内同目录下是否有不一致，如果有不一样的文件则同步文件过去，如果一致则不更新。
2.2. xsync 脚本
-
在 bin 目录下创建 xsync；
cd /bin & vim xsync -
编辑脚本；
#!/bin/sh # 获取输入参数个数，如果没有参数，直接退出 pcount=$# if((pcount!=4)); then echo Usage: $0 filename servername startno endno exit; fi # 获取文件名称 p1=$1 fname=`basename $p1` echo fname=$fname # 获取上级目录到绝对路径 pdir=`cd -P $(dirname $p1); pwd` echo pdir=$pdir # 获取当前用户名称 user=`whoami` # 获取hostname及起止号 slave=$2 startline=$3 endline=$4 # 循环 for((host=$startline; host<=$endline; host++)); do echo $pdir/$fname $user@$slave$host:$pdir echo ==================$slave$host================== rsync -rvl $pdir/$fname $user@$slave$host:$pdir done -
修改权限
chmod 777 xsync
2.3. xsync 使用
-
使用示例：
[root@hadoop-1 home]# sudo xsync 1.txt slave 1 2 fname=1.txt pdir=/home /home/1.txt root@slave1:/home ==================slave1================== sending incremental file list 1.txt sent 90 bytes received 35 bytes 83.33 bytes/sec total size is 2 speedup is 0.02 /home/1.txt root@slave2:/home ==================slave2================== sending incremental file list 1.txt sent 90 bytes received 35 bytes 250.00 bytes/sec total size is 2 speedup is 0.02
3. javapsall 脚本
-
建立 jps 软链接
ln -s /usr/local/jdk1.8/bin/jps /usr/local/bin/jps -
在
/usr/local/bin路径下编辑javapsall脚本for host in hadoop102 hadoop103 hadoop104 do echo "========== $host =========" ssh $host "jps" | grep -v Jps done -
修改 javapsall 权限
chmod 777 javapsall -
测试结果
[root@hadoop102 bin]# javapsall ========== hadoop102 ========= ========== hadoop103 ========= ========== hadoop104 =========
4. xcall 集群内执行命令
4.1. 编辑 xcall
cd /usr/local/bin
vim xcall
#!/bin/bash
pcount=$#
if((pcount==0));then
echo "没有参数！";
exit;
fi
echo -------hadoop102------
$@
for((host=2; host<=4; host++)); do
echo -------hadoop10$host------
ssh hadoop10$host $@
done
chmod 777 xcall
4.2. xcall 使用示例
xcall jps
-------hadoop102------
13586 DataNode
13994 Jps
13933 JobHistoryServer
13455 NameNode
-------hadoop102------
13586 DataNode
14009 Jps
13933 JobHistoryServer
13455 NameNode
-------hadoop103------
5040 Jps
4703 NodeManager
4575 ResourceManager
-------hadoop104------
3408 SecondaryNameNode
3443 Jps
5. Hadoop 集群启动命令
5.1. 编辑 myhadoop.sh 文件
-
在
/usr/local/bin下创建文件#!/bin/bash # 判断参数个数 if [ $# -ne 1 ];then echo "need one param, but given $#" fi # 操作hadoop case $1 in "start") echo " ========== 启动hadoop集群 ========== " echo ' ---------- 启动 hdfs ---------- ' ssh hadoop102 "/usr/local/hadoop-3.1.3/sbin/start-dfs.sh" echo ' ---------- 启动 yarn ---------- ' ssh hadoop103 "/usr/local/hadoop-3.1.3/sbin/start-yarn.sh" echo ' ---------- 启动 historyserver ---------- ' ssh hadoop102 "/usr/local/hadoop-3.1.3/sbin/mr-jobhistory-daemon.sh start historyserver" ;; "stop") echo " ========== 关闭hadoop集群 ========== " echo ' ---------- 关闭 historyserver ---------- ' ssh hadoop102 "/usr/local/hadoop-3.1.3/sbin/mr-jobhistory-daemon.sh stop historyserver" echo ' ---------- 关闭 hdfs ---------- ' ssh hadoop102 "/usr/local/hadoop-3.1.3/sbin/stop-dfs.sh" echo ' ---------- 关闭 yarn ---------- ' ssh hadoop103 "/usr/local/hadoop-3.1.3/sbin/stop-yarn.sh" ;; *) echo "Input Param Error ..." ;;
5.2. 集群启动关闭使用示例
-
启动集群
myhadoop.sh start ========== 启动hadoop集群 ========== ---------- 启动 hdfs ---------- Starting namenodes on [hadoop102] Last login: Fri May 13 23:23:13 CST 2022 Starting datanodes Last login: Fri May 13 23:23:25 CST 2022 Starting secondary namenodes [hadoop102] Last login: Fri May 13 23:23:28 CST 2022 ---------- 启动 yarn ---------- Starting resourcemanager Last login: Fri May 13 23:23:18 CST 2022 Starting nodemanagers Last login: Fri May 13 23:23:38 CST 2022 ---------- 启动 historyserver ---------- WARNING: Use of this script to start the MR JobHistory daemon is deprecated. WARNING: Attempting to execute replacement "mapred --daemon start" instead. -
关闭集群
myhadoop.sh stop ========== 关闭hadoop集群 ========== ---------- 关闭 historyserver ---------- WARNING: Use of this script to stop the MR JobHistory daemon is deprecated. WARNING: Attempting to execute replacement "mapred --daemon stop" instead. ---------- 关闭 hdfs ---------- Stopping namenodes on [hadoop102] Last login: Fri May 13 23:07:27 CST 2022 from 183.14.30.31 on pts/1 Stopping datanodes Last login: Fri May 13 23:23:12 CST 2022 Stopping secondary namenodes [hadoop102] Last login: Fri May 13 23:23:12 CST 2022 ---------- 关闭 yarn ---------- Stopping nodemanagers Last login: Fri May 13 22:24:28 CST 2022 from 183.14.30.31 on pts/0 Stopping resourcemanager Last login: Fri May 13 23:23:16 CST 2022
