History

Parallel NFS (pNFS)

Introduction

Parallel NFS (pNFS) is a part of the NFS v4.1 standard that allows clients to access storage devices in parallel. The pNFS architecture provides greater performance and scalability when compared to legacy NFS technology

How is pNFS different from legacy NFS?

Clients are given the ability to access the data servers directly.
Data can be striped across multiple data servers.
Easily scalable for better performance and higher throughput.

Advantages

High Performance : Provide scalable parallel access to files distributed among multiple servers.
Horizontal Scalability : Reduce the resource limitation of the single NFS server serving all of the files in an exported file system.
Separation of Data and Metadata: Capability to move the data around and dynamically notify the client of the location.

pNFS on Nexenta

Currently the pNFS functionality is based out of ON-gate build 121. We've used this release to build pNFS powered Nexenta Core Platform.

These sources and other related downloads are available here.

Instructions for setting up pNFS on Nexenta Core Platform

There are 3 main components of a pNFS configuration: Client, Metadata Server (MDS) and Data Server (DS).

Currently, the DS and the MDS must be setup on separate machines. For the basic set-up its recommended that you have 3 machines all running NCP-pNFS, one to setup as Client, one as MDS, and one as DS.

The following instructions help you to create a set-up and test pNFS.

Setting up Meta Data Server(MDS)

Create a share path.

#share [pathname]

Eg:
#share /export

NOTE: This is not persistent over reboots.

Setting up Data Server(DS)

Share a filesystem

#sharemgr create -P nfs [share-group]
#sharemgr add-share -s [share-target] [share-group]

Eg:
#sharemgr create -P nfs p-group
#sharemgr add-share -s /export p-group

NOTE: The above is done simply to ensure that NFS server service is up and running. This is because, the data server SMF service (svc:/network/dserv:default) depends on the NFS server service (svc:/network/nfs/server:default).

The Data Server is managed using dservadm utility

Add the address of MDS for use by DS

#dservadm addmds [IP-addr.8.1]

Eg:
#dservadm addmds 10.1.1.7.8.1

.8.1 happens to be the port on which the DS listens. It is necessary to specify this.

dservadm does not have a manpage yet. You can run dservadm --help.

Make an entry into the /etc/hosts file to add the IP-address, hostname of DS (Assume that DS has the IP: 10.1.1.241)

#cat /etc/hosts
::1                 localhost
127.0.0.1       localhost
127.0.0.1       ds      ds.mydomain     loghost
10.1.1.241    ds      ds.mydomain     loghost

Now, create a pnfs dataset:

#zpool create [pool] [vdev]
#zfs create -t pnfsdata [pnfs-dataset]

Eg:
#zpool create ppool /dev/dsk/c1t1d0
#zfs create -t pnfsdata ppool/pdata

NOTE: There should be at least one pNFS dataset per data server, so if you plan on setting up multiple instances on your data server, be sure to create at least one pNFS dataset per server.

Add the dataset to the list of datasets managed by the Data Server.

#dservadm addstor ppool/pdata

Start the dserv service and check if its online!

#dservadm enable
#svcs dserv
    STATE          STIME        FMRI
    online         20:45:24    svc:/network/dserv:default

Setting up the client

Mounting the filesystem from the MDS on the client is as simple as:

#mount -F nfs -o vers=4 [mds:/path] [mount point]

Eg:
#mount -F nfs -o vers=4 10.1.1.7:/export /mnt/pnfs

Current Limitation

Deletion of files or directories from the client machine or the MDS does not deallocate the unused blocks on the Data Server. Eventually this take up all the storage space on the Data Server, if left unchecked. We're looking into the reason for this issue, and figuring out a workaround.

Help

Post queries in our Forum.
IRC: Join us on #nexenta on Freenode.

Also available in: HTML TXT