The GPU cluster is provisioned with a set of two shared filesystems mounted on the head node as well as on all the GPU compute nodes as follows :
Filesystem Size Used Avail Use% Mounted on /dev/gpfs 12T 45G 12T 1% /GPU sn8-ib@o2ib:/lustre 7.9T 5.7G 7.5T 1% /LUSTRE
The “/GPU” filesystem of 12TB is served by a set of four (4) GPFS NSD Servers in a GPFS storage cluster. This mount is shared between all the nodes (hn, cn1, cn2,cn3,cn4,cn5). System administrators should refer to the seperate Wiki namespace on GPFS for configuration, management and operations of the GPFS cluster.
The “/LUSTRE” filesystem mount is a 7.9TB shared filesystem served by a Lustre cluster running on a single shared Lustre MDS/MGS, plus a set of 3 x ODS servers, serving a total of 9 Lustre OST's (3 per ODS). System administrators can refer to the Wiki namespace on Lustre for configuration, management and operations of the Lustre cluster.
Connectivity between GPFS clients, Lustre clients and the respective storage cluster servers uses 40Gbps QDR Infiniband, with GPFS using Direct Memory Access (RDMA) over Infiniband.