acelab:nfs_configuration
On the Host servers [nfs01 and nfs02]
Device Mapper Multipathing Devices (DMMP)
- Scan for virtual disks
- Display the multipath device topology
- Create a partition on a multipath device node
- Add a partition to Device Mapper(DM)
- Crate a file systems on a Device Mapper partition
- Mount a Device mapper partition
1. Install all packages that are needed
[root@nfs02 ~]# yum install device-mapper device-mapper-devel device-mapper-multipath kpartx [root@nfs02 ~]# yum install sg3_utils sg3_utils-devel
2. Start multipath service
[root@nfs02 ~]# service multipathd start Starting multipathd daemon: [ OK ] [root@nfs02 ~]# chkconfig multipathd on [root@nfs02 ~]#
3. Scan for devices and partitions
[root@nfs02 ~]# rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 1 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 2 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 4 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 7 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 8 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning host 9 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 9 0 0 0 ...
OLD: Host: scsi9 Channel: 00 Id: 00 Lun: 00
Vendor: PLDS Model: DVD-ROM DU-8D5LH Rev: 3D51
Type: CD-ROM ANSI SCSI revision: 05
Scanning host 10 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 10 2 0 0 ...
OLD: Host: scsi10 Channel: 02 Id: 00 Lun: 00
Vendor: DELL Model: PERC H330 Mini Rev: 4.27
Type: Direct-Access ANSI SCSI revision: 05
Scanning host 11 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs
Scanning for device 11 0 0 0 ...
NEW: Host: scsi11 Channel: 00 Id: 00 Lun: 00
Vendor: DELL Model: MD34xx Rev: 0825
Type: Direct-Access ANSI SCSI revision: 05
Scanning for device 11 0 0 31 ...
OLD: Host: scsi11 Channel: 00 Id: 00 Lun: 31
Vendor: DELL Model: Universal Xport Rev: 0825
Type: Direct-Access ANSI SCSI revision: 05
Scanning for device 11 0 0 1 ...
OLD: Host: scsi11 Channel: 00 Id: 00 Lun: 01
Vendor: DELL Model: MD34xx Rev: 0825
Type: Direct-Access ANSI SCSI revision: 05
Scanning for device 11 0 0 2 ...
NEW: Host: scsi11 Channel: 00 Id: 00 Lun: 02
Vendor: DELL Model: MD34xx Rev: 0825
Type: Direct-Access ANSI SCSI revision: 05
Scanning for device 11 0 0 3 ...
NEW: Host: scsi11 Channel: 00 Id: 00 Lun: 03
Vendor: DELL Model: MD34xx Rev: 0825
Type: Direct-Access ANSI SCSI revision: 05
Scanning for device 11 0 1 0 ...
NEW: Host: scsi11 Channel: 00 Id: 01 Lun: 00
Vendor: DELL Model: MD34xx Rev: 0825
Type: Direct-Access ANSI SCSI revision: 05
Scanning for device 11 0 1 31 ...
OLD: Host: scsi11 Channel: 00 Id: 01 Lun: 31
Vendor: DELL Model: Universal Xport Rev: 0825
Type: Direct-Access ANSI SCSI revision: 05
Scanning for device 11 0 1 1 ...
OLD: Host: scsi11 Channel: 00 Id: 01 Lun: 01
Vendor: DELL Model: MD34xx Rev: 0825
Type: Direct-Access ANSI SCSI revision: 05
Scanning for device 11 0 1 2 ...
NEW: Host: scsi11 Channel: 00 Id: 01 Lun: 02
Vendor: DELL Model: MD34xx Rev: 0825
Type: Direct-Access ANSI SCSI revision: 05
Scanning for device 11 0 1 3 ...
NEW: Host: scsi11 Channel: 00 Id: 01 Lun: 03
Vendor: DELL Model: MD34xx Rev: 0825
Type: Direct-Access ANSI SCSI revision: 05
6 new or changed device(s) found.
[11:0:0:0]
[11:0:0:2]
[11:0:0:3]
[11:0:1:0]
[11:0:1:2]
[11:0:1:3]
0 remapped or resized device(s) found.
0 device(s) removed.
[root@nfs02 ~]#
4. Display device mappings
[root@nfs02 ~]# multipath -ll mpathe (3600a098000b6278f00000240599ffeea) dm-3 DELL,MD34xx size=680G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw |-+- policy='round-robin 0' prio=14 status=active | `- 11:0:0:3 sde 8:64 active ready running `-+- policy='round-robin 0' prio=9 status=enabled `- 11:0:1:3 sdi 8:128 active ready running mpathd (3600a098000b5c042000003c1599fff37) dm-2 DELL,MD34xx size=200G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw |-+- policy='round-robin 0' prio=14 status=active | `- 11:0:1:2 sdh 8:112 active ready running `-+- policy='round-robin 0' prio=9 status=enabled `- 11:0:0:2 sdd 8:48 active ready running mpathc (3600a098000b5c042000003bf599ffed2) dm-0 DELL,MD34xx size=1.5T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw |-+- policy='round-robin 0' prio=14 status=active | `- 11:0:1:0 sdf 8:80 active ready running `-+- policy='round-robin 0' prio=9 status=enabled `- 11:0:0:0 sdb 8:16 active ready running mpathf (3600a098000b6278f0000023e599ffe83) dm-1 DELL,MD34xx size=2.5T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw |-+- policy='round-robin 0' prio=14 status=active | `- 11:0:0:1 sdc 8:32 active ready running `-+- policy='round-robin 0' prio=9 status=enabled `- 11:0:1:1 sdg 8:96 active ready running
5. Grab the image of nfs02and reboot nfs01 just to check if all is well.
[root@nfs01 ~]# multipath -ll mpathe (3600a098000b6278f00000240599ffeea) dm-3 DELL,MD34xx size=680G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw |-+- policy='round-robin 0' prio=14 status=active | `- 11:0:0:3 sde 8:64 active ready running `-+- policy='round-robin 0' prio=9 status=enabled `- 11:0:1:3 sdi 8:128 active ready running mpathd (3600a098000b5c042000003c1599fff37) dm-2 DELL,MD34xx size=200G features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw |-+- policy='round-robin 0' prio=14 status=active | `- 11:0:1:2 sdh 8:112 active ready running `-+- policy='round-robin 0' prio=9 status=enabled `- 11:0:0:2 sdd 8:48 active ready running mpathc (3600a098000b5c042000003bf599ffed2) dm-0 DELL,MD34xx size=1.5T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw |-+- policy='round-robin 0' prio=14 status=active | `- 11:0:1:0 sdf 8:80 active ready running `-+- policy='round-robin 0' prio=9 status=enabled `- 11:0:0:0 sdb 8:16 active ready running mpathf (3600a098000b6278f0000023e599ffe83) dm-1 DELL,MD34xx size=2.5T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw |-+- policy='round-robin 0' prio=14 status=active | `- 11:0:0:1 sdc 8:32 active ready running `-+- policy='round-robin 0' prio=9 status=enabled `- 11:0:1:1 sdg 8:96 active ready running [root@nfs01 ~]#
6. Storage Array Virtual Disks - Using Dell Storage Manager
Disk /dev/mapper/mpathc: 1.5 TB APPS Disk /dev/mapper/mpathd: 200 GB PBS_SHARED Disk /dev/mapper/mpathe: 680 GB CM_SHARED Disk /dev/mapper/mpathf: 2.5 TB HOME
7. Install “parted” if not already installed
[root@nfs01 ~]# yum install parted parted-devel
8. Create partitions
[root@nfs01 ~]# parted /dev/mapper/mpathc GNU Parted 2.1 Using /dev/mapper/mpathc Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel GPT (parted) mkpart primary xfs 0% 100% (parted) print Model: Linux device-mapper (multipath) (dm) Disk /dev/mapper/mpathc: 1649GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 1049kB 1649GB 1649GB primary (parted) quit Information: You may need to update /etc/fstab. [root@nfs01 ~]#
9. Create a XFS File system from the partition
[root@nfs01 ~]# mkfs.xfs -L APPS /dev/mapper/mpathcp1
meta-data=/dev/mapper/mpathcp1 isize=256 agcount=4, agsize=100663168 blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=402652672, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=196607, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@nfs01 ~]#
10. Crate partitions and file system for all virtual disks [PBS_SHARED,CM_SHARED,HOME]
11. Mount on /etc/fstab using LABELS - this is just for testing
[root@nfs01 ~]# cat /etc/fstab |grep LABEL LABEL=APPS /apps xfs defaults 0 0 LABEL=PBS_SHARED /pbsshared xfs defaults 0 0 LABEL=CM_SHARED /cmshared xfs defaults 0 0 LABEL=HOME /nfshome xfs defaults 0 0 [root@nfs01 ~]# mkdir /apps /pbsshared /cmshared /nfshome [root@nfs01 ~]# mount -a [root@nfs01 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 20G 4.5G 16G 23% / none 7.8G 0 7.8G 0% /dev/shm /dev/sda6 891G 33M 891G 1% /local /dev/sda3 2.0G 320M 1.7G 16% /tmp /dev/sda2 6.0G 300M 5.7G 5% /var master:/cm/shared 342G 68G 275G 20% /cm/shared master:/home 342G 68G 275G 20% /home /dev/mapper/mpathcp1 1.5T 33M 1.5T 1% /apps /dev/mapper/mpathdp1 200G 33M 200G 1% /pbsshared /dev/mapper/mpathep1 680G 33M 680G 1% /cmshared /dev/mapper/mpathfp1 2.5T 33M 2.5T 1% /nfshome [root@nfs01 ~]#
12. Add mount points to fsmounts on device nfs01 using bright
[bright1->device[nfs01]->fsmounts]% add /apps [bright1->device*[nfs01*]->fsmounts*[/apps*]]% set device /dev/mapper/mpathcp1 [bright1->device*[nfs01*]->fsmounts*[/apps*]]% set filesystem xfs [bright1->device*[nfs01*]->fsmounts*[/apps*]]% commit [bright1->device[nfs01]->fsmounts[/apps]]% exit [bright1->device[nfs01]->fsmounts]% add /pbsshared [bright1->device*[nfs01*]->fsmounts*[/pbsshared*]]% set device /dev/mapper/mpathdp1 [bright1->device*[nfs01*]->fsmounts*[/pbsshared*]]% set filesystem xfs [bright1->device*[nfs01*]->fsmounts*[/pbsshared*]]% commit [bright1->device[nfs01]->fsmounts[/pbsshared]]% exit [bright1->device[nfs01]->fsmounts]% add /cmshared [bright1->device*[nfs01*]->fsmounts*[/cmshared*]]% set device /dev/mapper/mpathep1 [bright1->device*[nfs01*]->fsmounts*[/cmshared*]]% set filesystem xfs [bright1->device*[nfs01*]->fsmounts*[/cmshared*]]% commit [bright1->device[nfs01]->fsmounts[/cmshared]]% exit [bright1->device[nfs01]->fsmounts]% add /nfshome [bright1->device*[nfs01*]->fsmounts*[/nfshome*]]% set device /dev/mapper/mpathfp1 [bright1->device*[nfs01*]->fsmounts*[/nfshome*]]% set filesystem xfs [bright1->device*[nfs01*]->fsmounts*[/nfshome*]]% commit [bright1->device[nfs01]->fsmounts[/nfshome]]% exit [bright1->device[nfs01]->fsmounts]% list Device Mountpoint (key) Filesystem --------------------------------- -------------------------------- ---------------- [nfs] none /dev/pts devpts [nfs] none /proc proc [nfs] none /sys sysfs [nfs] none /dev/shm tmpfs [nfs] $localnfsserver:/cm/shared /cm/shared nfs [nfs] $localnfsserver:/home /home nfs /dev/mapper/mpathcp1 /apps xfs /dev/mapper/mpathdp1 /pbsshared xfs /dev/mapper/mpathep1 /cmshared xfs /dev/mapper/mpathfp1 /nfshome xfs [bright1->device[nfs01]->fsmounts]%
13. The above step does the following (as soon as you commit
[root@nfs01 ~]# df -h|grep path /dev/mapper/mpathcp1 1.5T 33M 1.5T 1% /apps /dev/mapper/mpathdp1 200G 33M 200G 1% /pbsshared /dev/mapper/mpathep1 680G 33M 680G 1% /cmshared /dev/mapper/mpathfp1 2.5T 33M 2.5T 1% /nfshome [root@nfs01 ~]# cat /etc/fstab |grep path /dev/mapper/mpathcp1 /apps xfs defaults 0 0 /dev/mapper/mpathdp1 /pbsshared xfs defaults 0 0 /dev/mapper/mpathep1 /cmshared xfs defaults 0 0 /dev/mapper/mpathfp1 /nfshome xfs defaults 0 0 [root@nfs01 ~]#
14. Set up nfsexports through bright
[bright1->device*[nfs01*]->fsexports*[/apps*]]% set hosts 172.20.0.0/24 [bright1->device*[nfs01*]->fsexports*[/apps*]]% set async yes [bright1->device*[nfs01*]->fsexports*[/apps*]]% set rootsquash no [bright1->device*[nfs01*]->fsexports*[/apps*]]% set write yes [bright1->device*[nfs01*]->fsexports*[/apps*]]% append hosts " 172.18.0.0/24" [bright1->device*[nfs01*]->fsexports*[/apps*]]% commit [bright1->device[nfs01]->fsexports[/cmshared]]% exit [bright1->device[nfs01]->fsexports]% clone /apps /pbsshared [bright1->device[nfs01]->fsexports[/pbsshared]]% set path /pbsshared [bright1->device*[nfs01*]->fsexports*[/pbsshared*]]% commit [bright1->device[nfs01]->fsexports]% clone /apps /cmshared [bright1->device[nfs01]->fsexports[/cmshared]]% set path /cmshared [bright1->device*[nfs01*]->fsexports*[/cmshared*]]% commit [bright1->device[nfs01]->fsexports]% clone /apps /nfshome [bright1->device[nfs01]->fsexports[/nfshome]]% set path /nfshome [bright1->device*[nfs01*]->fsexports*[/nfshome*]]% commit Produces ... [root@nfs01 ~]# cat /etc/exports # This section of this file was automatically generated by cmd. Do not edit manually! # BEGIN AUTOGENERATED SECTION -- DO NOT REMOVE /apps 172.20.0.0/24(rw,no_root_squash,async) 172.18.0.0/24(rw,no_root_squash,async) /pbsshared 172.20.0.0/24(rw,no_root_squash,async) 172.18.0.0/24(rw,no_root_squash,async) /cmshared 172.20.0.0/24(rw,no_root_squash,async) 172.18.0.0/24(rw,no_root_squash,async) /nfshome 172.20.0.0/24(rw,no_root_squash,async) 172.18.0.0/24(rw,no_root_squash,async) # END AUTOGENERATED SECTION -- DO NOT REMOVE [root@nfs01 ~]#
15. Mount the fsmounts point [on all node categories]
[bright1->category]% foreach compute iml login scheduler (fsmounts; add /home; set device nfs01.ib.cluster:/nfshome; set filesystem nfs; commit) [bright1->category]% foreach compute iml login scheduler (fsmounts; add /cm/shared; set device nfs01.ib.cluster:/cmshared; set filesystem nfs; commit) [bright1->category]% foreach compute iml login scheduler (fsmounts; add /apps; set device nfs01.ib.cluster:/apps; set filesystem nfs; commit) [bright1->category]% foreach scheduler (fsmounts; add /pbsshared; set device nfs01.ib.cluster:/pbsshared; set filesystem nfs; commit) [bright1->category]%
/var/www/wiki/data/pages/acelab/nfs_configuration.txt · Last modified: 2018/07/27 19:52 by smasoka
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
