mkfs.lustre common problems (on going)

When installing lustre it is necessary to generate the storage targets, this is completed with mkfs.lustre, normally this happens without issue, however from time to time this isn’t the case. The following issues are ones I’ve seen and solutions are provided.

One of the most common issues is seen below (in this case, while we were trying to generate a metadata target):

mkfs.lustre --mdt --mgs --index 0 --fsname=lustre /dev/loop0

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters:

checking for existing Lustre data: not found
device size = 1024MB
formatting backing filesystem ldiskfs on /dev/loop0
        target name  lustre:MDT0000
        4k blocks     262144
        options        -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000  -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/loop0 262144
mkfs.lustre: Unable to mount /dev/loop0: No such device
Is the ldiskfs module available?

mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with 19 (No such device)

As the error above clearly indicates, ldiskfs module isn’t loaded, usually this means you forgot to load the lustre module, or worse, didn’t install the lustre kernel.

Another error commonly seen, especially on more modern systems is one we see below:

mkfs.lustre --mdt --mgs --index 0 --fsname=lustre /dev/loop0

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters:

checking for existing Lustre data: not found
device size = 1024MB
formatting backing filesystem ldiskfs on /dev/loop0
        target name  lustre:MDT0000
        4k blocks     262144
        options        -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000  -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/loop0 262144
mkfs.lustre: Can't make configs dir /tmp/mnt0tovaB/CONFIGS (Permission denied)

mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with -1 (Unknown error 18446744073709551615)

The above error can indicate an issue with the e2fsprogs version installed, try updating it. Another possible cause can be the presents of selinux and restrictive permissions. The simplest solution is to correct the permissions problem with selinux, or disable it.

What is Lustre?

From the Wikipedia article on Lustre File Systems:

Lustre is a type of parallel distributed file system, generally used for large-scale cluster computing. The name Lustre is a portmanteau word derived from Linux and cluster.[3] Lustre file system software is available under the GNU General Public License (version 2 only) and provides high performance file systems for computer clusters ranging in size from small workgroup clusters to large-scale, multi-site clusters.

Because Lustre file systems have high performance capabilities and open licensing, it is often used in supercomputers. At one time, six of the top 10 and more than 60 of the top 100 supercomputers in the world have Lustre file systems in them, including the world’s #2 ranked TOP500supercomputer, Titan in 2013.[4]

Lustre file systems are scalable and can be part of multiple computer clusters with tens of thousands of client nodes, tens of petabytes (PB) of storage on hundreds of servers, and more than a terabyte per second (TB/s) of aggregate I/O throughput.[5][6] This makes Lustre file systems a popular choice for businesses with large data centers, including those in industries such as meteorology, simulation, oil and gas, life science, rich media, and finance.[7]