hadoop hbase集群环境信息整理
- 1、硬件及系统版本信息
系统 |
Linux version 2.6.32-431.el6.x86_64 |
cpu |
Intel x86_64 2.3GHz 40核(逻辑) |
内存 |
256G |
硬盘 |
共12块SAS,每块5.5T |
网卡 |
1000Mb/s,网络延时<0.5ms |
2、机器列表
主机 |
x015015 |
x015016 |
x015017 |
x015018 |
x015026 |
x015027 |
x015028 |
x015029 |
x015030 |
x015031 |
x015032 |
y015015 |
y015016 |
y015017 |
y015018 |
3、系统参数设置
系统配置项 | 值 |
最大文件数 |
655350 |
最大进程数 |
655350 |
swap分区 |
关闭 |
预读缓存区大小 |
2048K |
系统磁盘IO调度算法 |
CFQ |
ntp同步 |
是 |
4、软件准备
软件 | 版本 |
JDK |
1.7.0_65-b17 |
zookeeper |
zookeeper-3.4.6 |
hadoop | 2.5.2 |
hbase | 1.1.2 |
5、安装用户、组
用户名 | hadoop |
所属组 | hero |
家目录 | /home/hadoop |
6、JVM配置
角色 | 堆大小 | JVM参数 |
NameNode | 32G |
-Xms32G -Xmx32G -XX:PermSize=256M -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=65 -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError -Xloggc:${hadoop.home}/hadoop-$hadoop_ident_string-$command-$hostname. gc.log |
DataNode |
16G |
基本配置同上 |
JournalNode |
2G |
基本配置同上 |
ZKFC |
2G |
基本配置同上 |
HMaster |
32G |
基本配置同上 |
RegionServer |
64G |
基本配置同上 |
7、hadoop core-site.xml
<configuration> <property> <name>dfs.permissions</name> <value>false</value> </property> <property> <name>fs.defaultFS</name> <value>hdfs://mgfscluster</value> </property> <property> <name>io.file.buffer.size</name> <value>1048576</value> </property> <property> <name>io.bytes.per.checksum</name> <value>131072</value> </property> <property> <name>hadoop.tmp.dir</name> <name>/home/hadoop/tmp</name> </property> <property> <name>ha.zookeeper.quorum</name> <value>node1:2181,node2:2181,node3:2181,node4:2181,node5:2181</value> </property> <property> <name>ha.zookeeper.parent-znode</name> <value>/mgfscluster/hadoop-ha</value> </property> <property> <name>fs.trash.interval</name> <value>4320</value> </property> <property> <name>fs.trash.checkpoint.interval</name> <value>60</value> </property> <property> <name>ipc.server.listen.queue.size</name> <value>2048</value> </property> <property> <name>file.stream-buffer-size</name> <value>1048576</value> </property> <property> <name>file.bytes-per-checksum</name> <value>131072</value> </property> <property> <name>file.blocksize</name> <value>134217728</value> </property> <property> <name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value> </property> </configuration> |
8、hadoop hdfs-site.xml
<configuration> <property> <name>dfs.nameservices</name> <value>mgfscluster</value> </property> <property> <name>dfs.ha.namenodes.mgfscluster</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.mgfscluster.nn1</name> <value>x015026:9100</value> </property> <property> <name>dfs.namenode.rpc-address.mgfscluster.nn2</name> <value>x015027:9100</value> </property> <property> <name>dfs.namenode.http-address.mgfscluster.nn1</name> <value>x015026:50071</value> </property> <property> <name>dfs.namenode.http-address.mgfscluster.nn2</name> <value>x015027:50071</value> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/home/hadoop/data/hdfs/journal/</value> </property> <property> <name>dfs.namenode.shared.edits.dir.mgfscluster.nn1</name> <value>qjournal://x015026:8485;x015027:8485;x015028:8485/mgfscluster</value> </property> <property> <name>dfs.namenode.shared.edits.dir.mgfscluster.nn2</name> <value>qjournal://x015026:8485;x015027:8485;x015028:8485/mgfscluster</value> </property> <property> <name>dfs.client.failover.proxy.provider.mgfscluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/hadoop/.ssh/id_rsa</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:///home/hadoop/data/hdfs/nn0</value> <final>true</final> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:///home/hadoop/data/hdfs/dd0,file:///home/hadoop/data/hdfs/dd1,file:///home/hadoop/data/hdfs/dd2,file:///home/hadoop/data/hdfs/dd3,file:///home/hadoop/data/hdfs/dd4,file:///home/hadoop/data/hdfs/dd5,file:///home/hadoop/data/hdfs/dd6,file:///home/hadoop/data/hdfs/dd7,file:///home/hadoop/data/hdfs/dd8,file:///home/hadoop/data/hdfs/dd9,file:///home/hadoop/data/hdfs/dd10,file:///home/hadoop/data/hdfs/dd11</value> <final>true</final> </property> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <property> <name>dfs.namenode.handler.count</name> <value>128</value> </property> <property> <name>dfs.datanode.handler.count</name> <value>128</value> </property> <property> <name>dfs.datanode.du.reserved</name> <value>53687091200</value> </property> <property> <name>dfs.blockreport.initialDelay</name> <value>120</value> </property> <property> <name>dfs.datanode.max.transfer.threads</name> <value>4096</value> </property> <property> <name>dfs.heartbeat.interval</name> <value>3</value> </property> <property> <name>dfs.namenode.stale.datanode.interval</name> <value>30</value> </property> <property> <name>dfs.blocksize</name> <value>134217728</value> </property> <property> <name>dfs.datanode.failed.volumes.tolerated</name> <value>2</value> </property> <property> <name>dfs.datanode.balance.bandwidthPerSec</name> <value>4194304</value> </property> <property> <name>dfs.stream-buffer-size</name> <value>131072</value> </property> <property> <name>dfs.bytes-per-checksum</name> <value>16384</value> </property> <property> <name>dfs.client-write-packet-size</name> <value>65536</value> </property>
</configuration> |
9、hbase hbase-site.xml
<configuration> <property> <name>hbase.master</name> <value>x015015:60000</value> </property> <property> <name>hbase.rootdir</name> <value>hdfs://mgfscluster/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>node1:2181,node2:2181,node3:2181,node4:2181,node5:2181</value> </property> <property> <name>hbase.zookeeper.property.dataDir</name> <value>/home/hadoop/data/hbase/zookeeper</value> </property> <property> <name>zookeeper.znode.parent</name> <value>/hbasecluster</value> </property> <property> <name>hbase.tmp.dir</name> <value>/home/hadoop/data/hbase/tmp</value> </property> <property> <name>hbase.fs.tmp.dir</name> <value>/home/hadoop/data/hbase/tmp/hbase-staging</value> </property> <property> <name>hbase.local.dir</name> <value>/home/hadoop/data/hbase/local</value> </property> <property> <name>hbase.master.logcleaner.ttl</name> <value>600000</value> </property> <property> <name>hbase.regionserver.logroll.period</name> <value>3600000</value> </property> <property> <name>hbase.regionserver.global.memstore.size</name> <value>0.4</value> </property> <property> <name>hbase.regionserver.global.memstore.size.lower.limit</name> <value>0.35</value> </property> <property> <name>hbase.regionserver.region.split.policy</name> <value>org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy</value> </property> <property> <name>hbase.regionserver.regionSplitLimit</name> <value>1600</value> </property> <property> <name>zookeeper.session.timeout</name> <value>120000</value> </property> <property> <name>hbase.normalizer.enabled</name> <value>true</value> </property> <property> <name>hbase.normalizer.period</name> <value>3600000</value> </property> <property> <name>hbase.server.thread.wakefrequency</name> <value>10000</value> </property> <property> <name>hbase.server.versionfile.writeattempts</name> <value>3</value> </property> <property> <name>hbase.hregion.memstore.flush.size</name> <value>134217728</value> </property> <property> <name>hbase.hregion.memstore.block.multiplier</name> <value>4</value> </property> <property> <name>hbase.hregion.memstore.mslab.enabled</name> <value>true</value> </property> <property> <name>hbase.hregion.max.filesize</name> <value>6442450944</value> </property> <property> <name>hbase.hregion.majorcompaction</name> <value>604800000</value> </property> <property> <name>hbase.hstore.compactionThreshold</name> <value>5</value> </property> <property> <name>hbase.hstore.flusher.count</name> <value>8</value> </property> <property> <name>hbase.hstore.blockingStoreFiles</name> <value>16</value> </property> <property> <name>hbase.hstore.blockingWaitTime</name> <value>30000</value> </property> <property> <name>hbase.hstore.compaction.min</name> <value>6</value> </property> <property> <name>hbase.hstore.compaction.max</name> <value>12</value> </property> <property> <name>hbase.hstore.compaction.min.size</name> <value>134217728</value> </property> <property> <name>hbase.hstore.compaction.ratio</name> <value>1.2F</value> </property> <property> <name>hbase.regionserver.thread.compaction.throttle</name> <value>2684354560</value> </property> <property> <name>hbase.hstore.compaction.kv.max</name> <value>100</value> </property> <property> <name>hbase.storescanner.parallel.seek.enable</name> <value>true</value> </property> <property> <name>hbase.storescanner.parallel.seek.threads</name> <value>10</value> </property> <property> <name>hfile.block.cache.size</name> <value>0.4</value> </property> <property> <name>hbase.rpc.timeout</name> <value>90000</value> </property> <property> <name>hbase.server.compactchecker.interval.multiplier</name> <value>1000</value> </property> <property> <name>hbase.security.authentication</name> <value>simple</value> </property> <property> <name>hbase.regionserver.storefile.refresh.period</name> <value>15000</value> </property> <property> <name>hbase.region.replica.replication.enabled</name> <value>true</value> </property> <property> <name>hbase.replication</name> <value>true</value> </property> <property> <name>hbase.ipc.warn.response.time</name> <value>3000</value> </property> <property> <name>hbase.ipc.warn.response.size</name> <value>10485760</value> </property> <property> <name>hbase.quota.enabled</name> <value>true</value> </property> <property> <name>hbase.regionserver.handler.count</name> <value>180</value> </property> <property> <name>hbase.snapshot.enabled</name> <value>true</value> </property> <property> <name>hbase.rest.port</name> <value>20550</value> </property> <property> <name>hbase.rest.info.port</name> <value>8085</value> </property> <property> <name>>hbase.rest.readonly</name> <value>false</value> </property> <property> <name>hbase.rest.threads.min</name> <value>2</value> </property> <property> <name>hbase.thrift.minWorkerThreads</name> <value>200</value> </property> <property> <name>hbase.thrift.info.port</name> <value>9095</value> </property> <property> <name>thrift.accept-backlog</name> <value>511</value> </property> </configuration> |
相关推荐
Hadoop hbase hive sqoop集群环境安装配置及使用文档
hadoop+hbase集群搭建 详细手册
hadoop集群配置流程以及用到的配置文件,hadoop2.8.4、hbase2.1.0、zookeeper3.4.12
Hadoop HA高可用集群搭建(Hadoop+Zookeeper+HBase)
新手Hadoop_hbase集群搭建,3台服务器节点Centos7,Hadoop2.7.4,HBase1.3.1,jdk8,zookpeer3.4.10 安装包也将上传,亲手搭建编写文档
基于haodoop 集群搭建hbase集群。(2台机器的情况。)
徐老师大数据培训Hadoop+HBase+ZooKeeper+Spark+Kafka+Scala+Ambari
数据仓库hadoop+zookeeper+hbase集群安装方法记录,自己搭建纯手写的记录。相关软件请自行下载
NULL 博文链接:https://zhengzhuangjie.iteye.com/blog/1408185
七月在线七月在线## Note, this file is written by cloud-init on first boot of an instance
包含安装文件下载安装文件下载http://pan.baidu.com/share/link?shareid=663788401&uk=1947167851 linux下载http://pan.baidu.com/share/link?shareid=670839034&uk=1947167851
hadoop+hbase分布式集群安装方法。
hadoop+hbase+hive集群搭建
通过本文档,你将掌握搭建hadoop和hbase的方法,深入浅出,易学上手快。
本文档详细介绍了如何用ZooKeeper和Hadoop、HBase搭建分布式大数据分析平台。
Hadoop_Hbase集群安装部署手册 本系统操作系统使用的是hadoop2.7.1,操作系统为centos 6.5 64位
Hadoop2.2+Zookeeper3.4.5+HBase0.96集群环境搭建
Hadoop2.7.3+HBase1.2.5+ZooKeeper3.4.6 搭建分布式集群环境详解。 详细介绍了如何搭建分布式集群环境。
docker安装 docker安装方法有多种,下面列举在Windows和Linux系统中的安装步骤:12 Windows系统中的安装方法: 对于Windows Server系统,可以使用Microsoft发布的PowerShell模块DockerMicrosoftProvider来安装...
自己根据部署的集群环境,整理的Hbase集群搭建教程,如果有错误的地方还请指出;