基于ha的大数据平台Zookeeper、Hadoop(HA)、hbase(HA)搭建过程(下)

  • A+
所属分类:linux实战
  1. 五、安装hadoop(五台机器)
  2. hadoop(hdfs)的HA(2个namenode之间主备切换),
  3. 主备切换用的是zookeeper来做active、stantby之间切换的。
  4. 所以安装hadoop集群之前 先安装zookeeper集群
  5. 1、下载
  6. [hadoop@manager1 ~]$ wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.7.5/hadoop-2.7.5.tar.gz
  7. 2、解压
  8. [hadoop@manager1 ~]$ tar zxvf hadoop-2.7.5.tar.gz
  9. 3、修改hadoop环境变量
  10. [hadoop@manager1 ~]$ tail -n2 .bashrc
  11. HADOOP_HOME=/home/hadoop/hadoop-2.7.5
  12. export HADOOP_HOME
  13. 4、使变量生效
  14. [hadoop@manager1 ~]$ source .bashrc
  15. [hadoop@manager1 ~]$ echo $HADOOP_HOME
  16. /home/hadoop/hadoop-2.7.5
  17. 5、相关配置
  18. [hadoop@manager1 ~]$ ll hadoop-2.7.5/etc/hadoop/*
  19. -rw-r--r-- 1 hadoop hadoop  4436 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/capacity-scheduler.xml
  20. -rw-r--r-- 1 hadoop hadoop  1335 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/configuration.xsl
  21. -rw-r--r-- 1 hadoop hadoop   318 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/container-executor.cfg
  22. -rw-r--r-- 1 hadoop hadoop   774 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/core-site.xml
  23. -rw-r--r-- 1 hadoop hadoop  3670 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/hadoop-env.cmd
  24. -rw-r--r-- 1 hadoop hadoop  4224 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/hadoop-env.sh
  25. -rw-r--r-- 1 hadoop hadoop  2598 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/hadoop-metrics2.properties
  26. -rw-r--r-- 1 hadoop hadoop  2490 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/hadoop-metrics.properties
  27. -rw-r--r-- 1 hadoop hadoop  9683 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/hadoop-policy.xml
  28. -rw-r--r-- 1 hadoop hadoop   775 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/hdfs-site.xml
  29. -rw-r--r-- 1 hadoop hadoop  1449 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/httpfs-env.sh
  30. -rw-r--r-- 1 hadoop hadoop  1657 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/httpfs-log4j.properties
  31. -rw-r--r-- 1 hadoop hadoop    21 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/httpfs-signature.secret
  32. -rw-r--r-- 1 hadoop hadoop   620 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/httpfs-site.xml
  33. -rw-r--r-- 1 hadoop hadoop  3518 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/kms-acls.xml
  34. -rw-r--r-- 1 hadoop hadoop  1527 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/kms-env.sh
  35. -rw-r--r-- 1 hadoop hadoop  1631 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/kms-log4j.properties
  36. -rw-r--r-- 1 hadoop hadoop  5540 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/kms-site.xml
  37. -rw-r--r-- 1 hadoop hadoop 11237 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/log4j.properties
  38. -rw-r--r-- 1 hadoop hadoop   951 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/mapred-env.cmd
  39. -rw-r--r-- 1 hadoop hadoop  1383 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/mapred-env.sh
  40. -rw-r--r-- 1 hadoop hadoop  4113 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/mapred-queues.xml.template
  41. -rw-r--r-- 1 hadoop hadoop   758 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/mapred-site.xml.template
  42. -rw-r--r-- 1 hadoop hadoop    10 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/slaves
  43. -rw-r--r-- 1 hadoop hadoop  2316 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/ssl-client.xml.example
  44. -rw-r--r-- 1 hadoop hadoop  2697 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/ssl-server.xml.example
  45. -rw-r--r-- 1 hadoop hadoop  2191 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/yarn-env.cmd
  46. -rw-r--r-- 1 hadoop hadoop  4567 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/yarn-env.sh
  47. -rw-r--r-- 1 hadoop hadoop   690 Dec 16 09:12 hadoop-2.7.5/etc/hadoop/yarn-site.xml
  48. 6、修改配置hadoop-env.sh(我这里ssh客户端端口是54321 所以这边需要修改 默认22的不需要修改)
  49. [hadoop@manager1 ~]$ vim /home/hadoop/hadoop-2.7.5/etc/hadoop/hadoop-env.sh
  50. 一共修改2个地方
  51. #注释掉1个 并修改
  52. #export JAVA_HOME=${JAVA_HOME}
  53. export JAVA_HOME=/usr/java/jdk1.8.0_151
  54. export HADOOP_SSH_OPTS="-p 54321"
  55. 7、修改core-site.xml(hadoop全局配置)这里会用到zookeeper的IP地址
  56. [hadoop@hadoop1 hadoop]$ cat core-site.xml
  57. <configuration>
  58. <!-- 这里的manager是逻辑名称 也就是可以自定义 -->
  59. <property>
  60. <name>fs.defaultFS</name>
  61. <value>hdfs://manager:9000</value>
  62. </property>
  63. <!--指定hadoop数据临时存放目录-->
  64. <property>
  65. <name>hadoop.tmp.dir</name>
  66. <value>/home/hadoop/hadoop-2.7.5/hdfs/tmp</value>
  67. </property>
  68. <!--指定zookeeper集群地址-->
  69. <property>
  70. <name>ha.zookeeper.quorum</name>
  71. <value>node1:2181,node2:2181,node3:2181</value>
  72. </property>
  73. </configuration>
  74. 8、创建tmp临时目录
  75. [hadoop@manager1 ~]$ mkdir -pv hadoop-2.7.5/hdfs/tmp
  76. mkdir: created directory ‘hadoop-2.7.5/hdfs’
  77. mkdir: created directory ‘hadoop-2.7.5/hdfs/tmp’
  78. 9、修改hdfs-site.xml(namenode、journalnode服务)
  79. <configuration>
  80. <!--指定DataNode存储block的副本数量。默认值是3个 我这里有3个slave 如果是2个slave 那就填写2 -->
  81. <property>
  82. <name>dfs.replication</name>
  83. <value>3</value>
  84. </property>
  85. <!-- 指定hdfs元数据存储的路径 -->
  86. <property>
  87. <name>dfs.namenode.name.dir</name>
  88. <value>file:/home/hadoop/hadoop-2.7.5/hdfs/name</value>
  89. </property>
  90. <!-- 指定hdfs数据存储的路径 -->
  91. <property>
  92. <name>dfs.datanode.data.dir</name>
  93. <value>file:/home/hadoop/hadoop-2.7.5/hdfs/data</value>
  94. </property>
  95. <!--需要和core-site.xml中的保持一致 -->
  96. <property>
  97. <name>dfs.nameservices</name>
  98. <value>manager</value>
  99. </property>
  100. <!--我这里有两个NameNode,分别是m1,m2这里也是逻辑名称 自定义 -->
  101. <property>
  102. <name>dfs.ha.namenodes.manager</name>
  103. <value>m1,m2</value>
  104. </property>
  105. <!-- m1和m2的http通信地址 -->
  106. <property>
  107. <name>dfs.namenode.http-address.manager.m1</name>
  108. <value>manager1:50070</value>
  109. </property>
  110. <property>
  111. <name>dfs.namenode.http-address.manager.m2</name>
  112. <value>manager2:50070</value>
  113. </property>
  114. <!-- 配置m1,m2的rpc通信端口 -->
  115. <property>
  116. <name>dfs.namenode.rpc-address.manager.m1</name>
  117. <value>manager1:8020</value>
  118. </property>
  119. <property>
  120. <name>dfs.namenode.rpc-address.manager.m2</name>
  121. <value>manager2:8020</value>
  122. </property>
  123. <!-- 指定namenode元数据存储在journalnode中的路径 -->
  124. <property>
  125. <name>dfs.namenode.shared.edits.dir</name>
  126. <value>qjournal://node1:8485;node2:8485;node3:8485/manager</value>
  127. </property>
  128. <!-- 指定JournalNode在本地磁盘存放数据的位置 -->
  129. <property>
  130. <name>dfs.journalnode.edits.dir</name>
  131. <value>/home/hadoop/hadoop-2.7.5/hdfs/journaldata</value>
  132. </property>
  133. <!--开启自动故障转移 -->
  134. <property>
  135. <name>dfs.ha.automatic-failover.enabled</name>
  136. <value>true</value>
  137. </property>
  138. <!-- 配置隔离机制为ssh 这个东西和zkfc客户端启动有关系-->
  139. <property>
  140. <name>dfs.ha.fencing.methods</name>
  141. <value>sshfence</value>
  142. </property>
  143. </configuration>
  144. 还要创建1个目录journaldata
  145. [hadoop@manager1 ~]$ mkdir -pv hadoop-2.7.5/hdfs/journaldata
  146. mkdir: created directory ‘hadoop-2.7.5/hdfs/journaldata’
  147. 10、修改mapred-site.xml(服务)
  148. 这个暂时没修改 没启服务
  149. 11、还有个yarn-site.xml(服务)
  150. 这个暂时没修改 没启服务
  151. 13、修改slave配置 我这里是三台slave
  152. [hadoop@manager1 ~]$ vim hadoop-2.7.5/etc/hadoop/slaves
  153. node1
  154. node2
  155. node3
  156. ======================配置暂时告一段落 接下来初始化zookeeper和namenode======================
  157. 14、再格式化zookeeper集群之前 是要先启动zookeeper集群的 上面我已经启动好了
  158. [hadoop@manager1 ~]$ cd hadoop-2.7.5/bin/
  159. [hadoop@manager1 bin]$ ./hdfs zkfc -formatZK
  160. 图1
  161. 15、同时在node1的zookeeper上已经看到有hadoop-ha
  162. [hadoop@node1 zookeeper-3.4.10]$ bin/zkCli.sh
  163. Connecting to localhost:2181
  164. 2018-02-11 14:56:21,150 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
  165. 2018-02-11 14:56:21,153 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=node1
  166. 2018-02-11 14:56:21,153 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_151
  167. 2018-02-11 14:56:21,155 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
  168. 2018-02-11 14:56:21,155 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8.0_151/jre
  169. 2018-02-11 14:56:21,155 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/hadoop/zookeeper-3.4.10/bin/../build/classes:/home/hadoop/zookeeper-3.4.10/bin/../build/lib/*.jar:/home/hadoop/zookeeper-3.4.10/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hadoop/zookeeper-3.4.10/bin/../lib/slf4j-api-1.6.1.jar:/home/hadoop/zookeeper-3.4.10/bin/../lib/netty-3.10.5.Final.jar:/home/hadoop/zookeeper-3.4.10/bin/../lib/log4j-1.2.16.jar:/home/hadoop/zookeeper-3.4.10/bin/../lib/jline-0.9.94.jar:/home/hadoop/zookeeper-3.4.10/bin/../zookeeper-3.4.10.jar:/home/hadoop/zookeeper-3.4.10/bin/../src/java/lib/*.jar:/home/hadoop/zookeeper-3.4.10/bin/../conf:.:/usr/java/jdk1.8.0_151/lib:/usr/java/jdk1.8.0_151/jre/lib:
  170. 2018-02-11 14:56:21,155 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
  171. 2018-02-11 14:56:21,155 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
  172. 2018-02-11 14:56:21,155 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
  173. 2018-02-11 14:56:21,156 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
  174. 2018-02-11 14:56:21,156 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
  175. 2018-02-11 14:56:21,156 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-693.el7.x86_64
  176. 2018-02-11 14:56:21,156 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=hadoop
  177. 2018-02-11 14:56:21,156 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/hadoop
  178. 2018-02-11 14:56:21,156 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/hadoop/zookeeper-3.4.10
  179. 2018-02-11 14:56:21,157 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@1a86f2f1
  180. 2018-02-11 14:56:21,187 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
  181. Welcome to ZooKeeper!
  182. JLine support is enabled
  183. 2018-02-11 14:56:21,300 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@876] - Socket connection established to localhost/127.0.0.1:2181, initiating session
  184. 2018-02-11 14:56:21,326 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x161837ba29c0001, negotiated timeout = 30000
  185. WATCHER::
  186. WatchedEvent state:SyncConnected type:None path:null
  187. [zk: localhost:2181(CONNECTED) 0] ls /
  188. [zookeeper, hadoop-ha]
  189. 16、启动journalnode集群(node1 node2 node3)
  190. 启动node1上的journalnode
  191. [hadoop@node1 ~]$ cd hadoop-2.7.5
  192. [hadoop@node1 sbin]$ ./hadoop-daemon.sh start journalnode
  193. starting journalnode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-journalnode-node1.out
  194. [hadoop@node1 hadoop-2.7.5]$ jps -l
  195. 17986 org.apache.hadoop.hdfs.qjournal.server.JournalNode
  196. 17030 org.apache.zookeeper.server.quorum.QuorumPeerMain
  197. 18078 sun.tools.jps.Jps
  198. 相关启动日志
  199. [hadoop@node1 ~]$ cd hadoop-2.7.5/logs/
  200. [hadoop@node1 logs]$ tail -f hadoop-hadoop-journalnode-node1.log
  201. 2018-02-11 14:59:47,043 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context journal
  202. 2018-02-11 14:59:47,043 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
  203. 2018-02-11 14:59:47,043 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
  204. 2018-02-11 14:59:47,094 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8480
  205. 2018-02-11 14:59:47,094 INFO org.mortbay.log: jetty-6.1.26
  206. 2018-02-11 14:59:47,940 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8480
  207. 2018-02-11 14:59:48,089 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 500
  208. 2018-02-11 14:59:48,099 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8485
  209. 2018-02-11 14:59:48,414 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
  210. 2018-02-11 14:59:48,415 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8485: starting
  211. 启动node2上的journalnode
  212. [hadoop@node2 ~]$  cd hadoop-2.7.5
  213. [hadoop@node2 hadoop-2.7.5]$ sbin/hadoop-daemon.sh start journalnode
  214. starting journalnode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-journalnode-node2.out
  215. [hadoop@node2 hadoop-2.7.5]$ jps -l
  216. 17009 org.apache.zookeeper.server.quorum.QuorumPeerMain
  217. 17908 org.apache.hadoop.hdfs.qjournal.server.JournalNode
  218. 17991 sun.tools.jps.Jps
  219. 相关日志
  220. [hadoop@node1 ~]$ cd hadoop-2.7.5/logs/
  221. [hadoop@node2 logs]$ tail -f hadoop-hadoop-journalnode-node2.log
  222. 2018-02-11 15:00:53,257 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context journal
  223. 2018-02-11 15:00:53,257 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
  224. 2018-02-11 15:00:53,257 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
  225. 2018-02-11 15:00:53,290 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8480
  226. 2018-02-11 15:00:53,290 INFO org.mortbay.log: jetty-6.1.26
  227. 2018-02-11 15:00:53,871 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8480
  228. 2018-02-11 15:00:54,042 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 500
  229. 2018-02-11 15:00:54,054 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8485
  230. 2018-02-11 15:00:54,361 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
  231. 2018-02-11 15:00:54,363 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8485: starting
  232. 启动node3上的journalnode
  233. [hadoop@node3 ~]$ cd hadoop-2.7.5
  234. [hadoop@node3 hadoop-2.7.5]$ sbin/hadoop-daemon.sh start journalnode
  235. starting journalnode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-journalnode-node3.out
  236. [hadoop@node3 hadoop-2.7.5]$ jps -l
  237. 17924 org.apache.hadoop.hdfs.qjournal.server.JournalNode
  238. 18005 sun.tools.jps.Jps
  239. 17034 org.apache.zookeeper.server.quorum.QuorumPeerMain
  240. 相关日志
  241. [hadoop@node1 ~]$ cd hadoop-2.7.5/logs/
  242. [hadoop@node3 logs]$ tail -f hadoop-hadoop-journalnode-node3.log
  243. 2018-02-11 15:01:17,778 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context journal
  244. 2018-02-11 15:01:17,779 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
  245. 2018-02-11 15:01:17,779 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
  246. 2018-02-11 15:01:17,809 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8480
  247. 2018-02-11 15:01:17,809 INFO org.mortbay.log: jetty-6.1.26
  248. 2018-02-11 15:01:18,767 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8480
  249. 2018-02-11 15:01:18,937 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 500
  250. 2018-02-11 15:01:18,948 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8485
  251. 2018-02-11 15:01:19,268 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
  252. 2018-02-11 15:01:19,269 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8485: starting
  253. 17、namenode出场了 格式化HDFS(在manager1上执行)
  254. [hadoop@manager1 ~]$ cd hadoop-2.7.5
  255. [hadoop@manager1 hadoop-2.7.5]$ bin/hdfs namenode -format
  256. 图2
  257. 18、比较下manager1和manager2的目录结构
  258. [hadoop@manager1 hadoop-2.7.5]$ tree hdfs/
  259. hdfs/
  260. ├── journaldata
  261. ├── name
  262. │   └── current
  263. │       ├── fsimage_0000000000000000000
  264. │       ├── fsimage_0000000000000000000.md5
  265. │       ├── seen_txid
  266. │       └── VERSION
  267. └── tmp
  268. 4 directories, 4 files
  269. 我们看下manager2就少了name目录
  270. [hadoop@manager2 hadoop-2.7.5]$ tree hdfs/
  271. hdfs/
  272. ├── journaldata
  273. └── tmp
  274. 2 directories, 0 files
  275. 其实格式化就是多了1个目录 里面有一些数据了
  276. 19、启动第一个namanode上面已经格式化好了
  277. [hadoop@manager1 ~]$ cd hadoop-2.7.5
  278. [hadoop@manager1 hadoop-2.7.5]$ sbin/hadoop-daemon.sh start namenode
  279. 相关日志
  280. [hadoop@manager1 hadoop-2.7.5]$ tail -f logs/hadoop-hadoop-namenode-manager1.log
  281. 相关进程
  282. [hadoop@manager1 hadoop-2.7.5]$ jps
  283. 5076 NameNode
  284. 5197 Jps
  285. 20、同步数据 启动第二个namenode
  286. [hadoop@manager2 ~]$ cd hadoop-2.7.5
  287. [hadoop@manager2 hadoop-2.7.5]$ bin/hdfs namenode -bootstrapStandby
  288. 图3
  289. 可以看到其实就是从m1那边拉取数据
  290. [hadoop@manager2 hadoop-2.7.5]$ tree hdfs/
  291. hdfs/
  292. ├── journaldata
  293. ├── name
  294. │   └── current
  295. │       ├── fsimage_0000000000000000000
  296. │       ├── fsimage_0000000000000000000.md5
  297. │       ├── seen_txid
  298. │       └── VERSION
  299. └── tmp
  300. 4 directories, 4 files
  301. 这样就和namenoe2和namenode1是一致的了
  302. 21、启动namenode2
  303. [hadoop@manager2 hadoop-2.7.5]$ sbin/hadoop-daemon.sh start namenode
  304. starting namenode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-namenode-manager2.out
  305. 相关日志
  306. [hadoop@manager2 hadoop-2.7.5]$ tail -f logs/hadoop-hadoop-namenode-manager2.log
  307. 相关进程
  308. [hadoop@manager2 hadoop-2.7.5]$ jps
  309. 18778 Jps
  310. 18671 NameNode
  311. 22、接下来启动datenode 其实是在manager1上去使用脚本去分别启动node1 node2 node3上的服务
  312. [hadoop@manager1 hadoop-2.7.5]$ sbin/hadoop-daemons.sh start datanode
  313. node2: starting datanode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-datanode-node2.out
  314. node3: starting datanode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-datanode-node3.out
  315. node1: starting datanode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-datanode-node1.out
  316. 这个过程需要ssh免密 上面已经配置好了 所以可以直接从manager1上直接去node1 node2 node3上分别启动服务
  317. 相关进程
  318. [hadoop@node1 hadoop-2.7.5]$ jps
  319. 17986 JournalNode
  320. 18757 Jps
  321. 17030 QuorumPeerMain
  322. 18670 DataNode
  323. [hadoop@node2 hadoop-2.7.5]$ jps
  324. 17009 QuorumPeerMain
  325. 18579 DataNode
  326. 17908 JournalNode
  327. 18677 Jps
  328. [hadoop@node3 hadoop-2.7.5]$ jps
  329. 18592 DataNode
  330. 18705 Jps
  331. 17924 JournalNode
  332. 17034 QuorumPeerMain
  333. 23、接下来把刚才的服务(datenode namenode journalnode)可以杀掉 我们使用start-dfs.sh这个脚本 把所有服务都启动起来
  334. [hadoop@manager1 sbin]$ sbin/start-dfs.sh
  335. 上述这个脚本会启动4个服务 分别是datenode namenode journalnode zkfc
  336. 其实看下脚本里的内容就可以看出来了 里面有分别启动各个服务的 前提是要配置好免密ssh
  337. 我们来看下效果
  338. [hadoop@manager1 hadoop-2.7.5]$ sbin/start-dfs.sh
  339. Starting namenodes on [manager1 manager2]
  340. manager2: starting namenode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-namenode-manager2.out
  341. manager1: starting namenode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-namenode-manager1.out
  342. node3: starting datanode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-datanode-node3.out
  343. node1: starting datanode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-datanode-node1.out
  344. node2: starting datanode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-datanode-node2.out
  345. Starting journal nodes [node1 node2 node3]
  346. node3: starting journalnode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-journalnode-node3.out
  347. node1: starting journalnode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-journalnode-node1.out
  348. node2: starting journalnode, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-journalnode-node2.out
  349. Starting ZK Failover Controllers on NN hosts [manager1 manager2]
  350. manager2: starting zkfc, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-zkfc-manager2.out
  351. manager1: starting zkfc, logging to /home/hadoop/hadoop-2.7.5/logs/hadoop-hadoop-zkfc-manager1.out
  352. 从这里可以看出启动顺序
  353. manager1(namenode) manager2(namenode)
  354. datanode-node3 datanode-node2 datanode-node1
  355. journalnode-node3 journalnode-node2 journalnode-node1
  356. zkfc-manager2 zkfc-manager1
  357. 最后1个必然是数据库hbase我这里还没搭建
  358. 分别看下5台机器的jps
  359. [hadoop@manager1 ~]$ jps
  360. 7025 DFSZKFailoverController
  361. 6696 NameNode
  362. 7118 Jps
  363. [hadoop@manager2 ~]$ jps
  364. 19846 DFSZKFailoverController
  365. 19725 NameNode
  366. 19934 Jps
  367. [hadoop@node1 ~]$ jps
  368. 19827 Jps
  369. 19507 QuorumPeerMain
  370. 19717 JournalNode
  371. 19613 DataNode
  372. [hadoop@node1 ~]$ jps
  373. 19622 JournalNode
  374. 19738 Jps
  375. 19420 QuorumPeerMain
  376. 19518 DataNode
  377. [hadoop@node3 ~]$ jps
  378. 19637 JournalNode
  379. 19767 Jps
  380. 19449 QuorumPeerMain
  381. 19533 DataNode
  382. 暂时还没弄 年后在搭建
  383. 六、安装hbase(五台机器)
  384. hbase集群是建立在hadoop集群上的 所以先搭建hadoop集群
  385. 1、下载
  386. [hadoop@master ~]$ wget http://mirrors.hust.edu.cn/apache/hbase/1.2.6/hbase-1.2.6-bin.tar.gz
  387. 2、
  388. 3、
  389. 七、参考
  390. http://hadoop.apache.org/docs/r2.7.5/
  391. https://mp.weixin.qq.com/s/oxfFMBH1ekupp6pLFAcG2Q
  392. http://blog.csdn.net/u011414200/article/details/50437356
  393. 基于ha的大数据平台Zookeeper、Hadoop(HA)、hbase(HA)搭建过程(下) 基于ha的大数据平台Zookeeper、Hadoop(HA)、hbase(HA)搭建过程(下) 基于ha的大数据平台Zookeeper、Hadoop(HA)、hbase(HA)搭建过程(下) 基于ha的大数据平台Zookeeper、Hadoop(HA)、hbase(HA)搭建过程(下) 基于ha的大数据平台Zookeeper、Hadoop(HA)、hbase(HA)搭建过程(下) 基于ha的大数据平台Zookeeper、Hadoop(HA)、hbase(HA)搭建过程(下)
  • 我的微信
  • 这是我的微信扫一扫
  • weinxin
  • 我的微信公众号
  • 我的微信公众号扫一扫
  • weinxin

说点什么

您将是第一位评论人!

提醒
avatar