Lustre DNE (Distributed Namespace) basics

What is Lustre DNE?

The Lustre Distributed Namespace (DNE) feature is a new technology developed for the Lustre File system. The feature it self allows for the distribution of file and directory metadata across multiple Lustre Metadata Targets (MDT). With this, you can now effectively scale your metadata load across a near unlimited number of targets. This both helps in reducing single metadata server hardware requirements as well as greatly expanding the maximum number of files your Lustre file system may contain.

The metadata structure appears as follows:
dne_remote_dir

In this example from Intel’s DNE proposal document

Here we see that the parent directory (which in this case is the root of the file system) spawns a subdirectory indexed against the secondary Metadata Target (MDT1).

How is Lustre DNE Setup / Enabled?

Lustre DNE was originally introduced in Lustre 2.4 and further refined in later versions. In this case my examples below are from a recent Lustre 2.5 installation.

Lustre DNE can be enabled on any Lustre file system greater than or equal to version 2.4.0.  To enable DNE, all one must do is create a new Metadata Target with an index greater than 0, and pointing to your Lustre management node (MGS). Multiple Metadata targets may exist on the same node, however it is not recommended as you will run into both CPU and memory contention and may not see much gains due to resource starvation (bus speed, etc).

Once formated and mounted, DNE will be available, that said, any clients mounted to the file system will need to re-mount the file system to make use of the new Metadata Target(s). This is because, as of currently, online DNE changes are not supported, however plans are to allow this in the future.

How do I create directories which will exist on different DNE targets?

Creating directories to point to different DNE targets (Metadata Targets) is quite simple. Once you’ve mounted all your MDT’s against your file system, and mounted (or remounted) your clients, you can now create directories which will target specific Metadata targets on specific Metadata servers. To do this, utilize the new lfs mkdir command:

# lfs mkdir -i 1 MDT1
# lfs mkdir -i 2 MDT2

The above command will generate two directories, MDT1 which utilizes metadata target index 1, and MDT2 which utilizes metadata target index 2.

Once complete, all new sub directories and files created within either MDT1 or MDT2 directories will exist on the specified actual metadata target (in this case targets with index numbers 1, and 2).

Files or directories moved out side of, say MDT1 into MDT2 will be moved from the metadata target 1, to metadata target 2, and so on.

How do I disable DNE or remove Metadata Targets I no longer use?

Disabling DNE and removing Metadata targets requires that you first move all file and directory metadata off of all metadata targets greater than 0. An example of this would be a file system structured like so:

(MDT0) ROOT /
(MDT1) ROOT / MDT1
(MDT1) ROOT / MDT1 / HOMES
(MDT1) ROOT / MDT1 / DATA
(MDT2) ROOT / MDT2
(MDT2) ROOT / MDT2 / SCRATCH
(MDT0) ROOT / USER_DATA

To disable MDT2 you will need to move SCRATCH and all associated sub directories and files into either ROOT/MDT1/. or ROOT/.

Once you have removed the data from the MDT2 DNE target you can unmount the provider for MDT2.

To completely disable DNE you will need to move all data on MDT1 and MDT2 DNE targets back to MDT0, in this case ROOT/. or ROOT/USER_DATA/.

In either case of MDT target removal a tunefs.lustre –writeconf is required against all file system targets (primary MGS, MDT, and all OST’s)

mkfs.lustre common problems (on going)

When installing lustre it is necessary to generate the storage targets, this is completed with mkfs.lustre, normally this happens without issue, however from time to time this isn’t the case. The following issues are ones I’ve seen and solutions are provided.

One of the most common issues is seen below (in this case, while we were trying to generate a metadata target):

mkfs.lustre --mdt --mgs --index 0 --fsname=lustre /dev/loop0

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters:

checking for existing Lustre data: not found
device size = 1024MB
formatting backing filesystem ldiskfs on /dev/loop0
        target name  lustre:MDT0000
        4k blocks     262144
        options        -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000  -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/loop0 262144
mkfs.lustre: Unable to mount /dev/loop0: No such device
Is the ldiskfs module available?

mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with 19 (No such device)

As the error above clearly indicates, ldiskfs module isn’t loaded, usually this means you forgot to load the lustre module, or worse, didn’t install the lustre kernel.

Another error commonly seen, especially on more modern systems is one we see below:

mkfs.lustre --mdt --mgs --index 0 --fsname=lustre /dev/loop0

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters:

checking for existing Lustre data: not found
device size = 1024MB
formatting backing filesystem ldiskfs on /dev/loop0
        target name  lustre:MDT0000
        4k blocks     262144
        options        -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000  -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/loop0 262144
mkfs.lustre: Can't make configs dir /tmp/mnt0tovaB/CONFIGS (Permission denied)

mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with -1 (Unknown error 18446744073709551615)

The above error can indicate an issue with the e2fsprogs version installed, try updating it. Another possible cause can be the presents of selinux and restrictive permissions. The simplest solution is to correct the permissions problem with selinux, or disable it.