Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

First I attached the two hard disk of 10 GB and 8 GB respectively as shown below :

To see it attached or not use command: fdisk -l or lsblk

As you see in the above my hard disk device name is :

1. /dev/sdb
2. /dev/sdc

First step is to create a LVM and attach to the hadoop slave for sharing their storage so that in future we can able to increase the size on the fly without unmount the folder.

Creating LVM

1. we need to create a physical volume (PV) of this two hard disk:

command : pvcreate 1st_device_name 2nd_device_name

Eg: pvcreate /dev/sdb /dev/sdc

If you want to display the details of PV you created use command : pvdispaly /dev/sdb /dev/sdc

2. Create a Volume Group (VG) using the above created PV :

command : vgcreate VG_Name /dev/sdb /dev/sdc

Eg: vgcreate vg /dev/sdb /dev/sdc

If you want to display the details of VG you created use command : vgdispaly VG_name

3. Create a LVM of above created Volume Group of any size that we wanted to share or contribute to the Name node or master node :

a> create LVM of 10 GB :

command : lvcreate — size 10G — name LV_name VG_name

Eg: lvcreate — size 10G — name lv vg

b> Format the partition of LVM we created :

command : mkfs.ext4 /dev/VG_name/LV_name

Eg: mkfs.ext4 /dev/vg/lv

4. Now we need to mount this created partition with the folder that we want to share :

command : mount /dev/VG_name/LV_name /folder_name

To check it is successfully mounted or not use command :

df -h

Now we will check in the hadoop master node that the storage that we created is successfully attached or not

command : hadoop hfsadmin -report (in master node)

Increasing the Size :

Now using the LVM concept we need to change the size of the storage from the data node o the fly without unmount the folder…..

First change the size of the LVM :

command : lvextend — size +5G /dev/VG_name/LV_name

Format the extended size :

command : resize2fs /dev/VG_name/LV_name

Now again check the hadoop master node that the size extended or not…

command : hadoop hdfsadmin -report

Hence the size extended on the fly …….

Thank you for reading !!!!