BDR and snapshot

 

********** Cluster failure scenarios and backup ----Aug 11 2020

How many disk failures can be tolerated in DD? if there a disk failure how will you jandle? --it will give warning in cloud era web UI

<=1/2 the total number of disks

example:DN-->slaves-->disk failure-->

open CDH UI--NN data dir and DN data dir-->instances-->role groups-->data node default groups--->change groups here  (/dfs/dn) remove disk from dirctory and restart DN

select DN which has disk failure and move the DN into grup

goto HDfs config-->serch for DATA-->DN data dirctories--->edit individual role groups disks-->remove the failed disks from DN grops --->save changes-->restart it

Inform to LINUX team regarding failed disk-->they will raise a tkt to vendor-->they will replace it--->config-->DN data dirctory -->add disks

If server gets crashed 

If entire RACK gets failed? ---due to RACK topology it will be effect

back end DBs-->DB team will enable MYSQL active replication

if entire Data centre failure--->temporory site loss (floods)  --->permanent site loss (fire) (create DR cluster called BDR -back and disaster recovery)

meta data failure in CM/ambari/hive/oozie?---mysql backups (mysql dump cmd) and mysql HA

user application erres?---> Hdfs dfs -rm -r -f -skiptrash /path of directory (delete data permanetly in hdfs)-->enable hadoop trash (make fs.trash.interval>0)

NOTE : moving to trash will work only through FS shell.

rach failure? due to rack topology no eefect

********** Disaster Recovery (when this DR will happen)

chennai data center (cluster a-->3 master -->10 slaves--2 edge node--3 kafka)--flood happen 

hyd data center (same cluster replicated in hyd) --active active clsuetr and Active passive cluster

How can you replicate data between 2 diff clusters?

tools---using -->HDFS , HBASE, HIVE,YARn,sqoop, (stores data in kafka topics--enable kafka mirror maker--replicate data from source to destination topics),spark,zoo-keeper

kafka->open CDH UI-->add role instances-->slave1.hadoop.comm:9092 ...2..3.. (same topic replicated to destination clsuter as well)---once the data moved it will be expired after 7 days

kafka-->console consumer--whter you are getting data in desticatin cluster or not (other cmd--console consumer lag checker)

HDFS--HIVE--Hbase (backups)-copy hdfs and hive hbae data from one cluster to other cluster (source to destination)---to copy from soucer to destination CDH--HDP no problem--(only pre-requisite is we need to have NN connectivity between differnt clusters)

there will be 1TB and Pb of DATA-->how can you copy-->HDFS has utility called--distcp(hadoop distcp hdfs://<active NN<fqdn>:8020 souce clsuet path--destinative path)--it will run YARN job (so you need to run this cmd in destination cluster)--it will use all resources--(max of 5min to copy 1TB data

how to increase the performance of distcp job-->in same cmd -->hadoop distcp -m 100/200/300--here only map job runs no reducer here (specifying number of mappers (based on numbers of mappers performance will incrase)

distcp -update/override -p(same permisions) (if you copy/hbase/data -source to destination) under /data there is 1 TB added--when new data got asdded source ,it will copy in destination cluster

i don't want to run dist cp manually--got o CDH UI-->replication schedules-->give sooure to destication cluster details

******HDFS snapshots

hbase snapshot--->snaphot <name space>:<table name>

clone snapshot 'snapshot name'

snapshot restore-------------------->CDH UI--->snapshot policies--shedule (daily)--save policy--(it will not run any map reduce job) eg :HIVE/hbase snapshots

*** HDFS snapshot --------August 12 2020

copy of directory at particular point of time (real time we don't use this (we only enable for imp dirs) if you enable snapshot you can't delete that dir)

** upgrade

1.OS --RHEL 7.7 (linux team will gpgrade)

2.java -- yum -y <java rpm pkg>

3.mysql

2.CDH (CM and CDH) CDH 5.15.2 to 5.16.1  ---CDH 6.x available(test cluster upgrded to 6.x) (now CDP 7.0 )--hdfs yarn zk hbase hive (impala--HIVE LLAP) (sentry--ranger) (kerberos--kerberos) (CM--ambari)

CDP 7.0 (2 types) CDP on cloud (they use aws) and CDP on Data center (physical servers)

pkgs comes with cdp?----all same except Ranger

CDH -end support by december 2020 (you need to upgrade 6.x version

https://www.cloudera.com/about/training.html

1000 -nodes

config tools---pupput,ansible(community version),ansible tower(enterprise version) --develped by redhat

monitoring hadoop clsster---nagios,graphana (mostly reliable on colud era manager) 

**** security (spark and imapala pending)