Infiniband Tips and Tricks

Below are some commands I’ve found useful when dealing with infiniband and related issues.

Determine port state on all ports available within the fabric

The command ibnetdiscover command will show you the state of each port in the the fabric. It’s a good idea when running this command (on any node) to redirect the output into a file (especially if you have a large number of ports)

Determine indivudual port information

The ibportstate command can provide additional port information once you have the LID (available from ibnetdiscover)


ibportstate -L <lid> query

Determine current node description

smpquery nodedesc 1

Enable / Disable a port

ibportstate -L <lid> -P <port> enable
ibportstate -L <lid> -P <port> disable

Windows 7 dropping connections to SMB services regularly

One thing that’s been constantly bugging me, as well as my wife is the fact that in general, Samba (SMB) connectivity from our Windows 7 machines is spotty at best. In general we can connect, and transfer data to and from our ZFS file server. However at seemingly random times the transfers fail, or the shares become unavailable. This is extremely frustrating especially when doing something like streaming music.

After some digging around I found the SessTimeout variable which is described as:

Determines the duration of the secondary delay used in calculating a time-out value for outstanding operations. If the redirector does not receive a response to an outstanding operation before the resulting time-out expires, it considers the operation to have failed. The value of the SessTimeout entry can be thought of as a margin for error. If there is an unexpected delay, the redirector permits the operation this extra time to complete.

Sounds promising…. So I popped up regedit on the windows workstations and added the DWORD entry SessTimeout in:


To a value of 300. After that no more timeout issues! Whoohoo! Finally!

Getting the Nvidia distribution drivers installed with Ubuntu 14.10 and possibly later

So do to the fact that my HTPC died, and I didn’t want to restore the OS from a backup, I opted instead to install a fresh (and reasonably modern) version of Kubuntu. Once installed I quickly ran into problems with the binary video driver installation. Yes, yes, I realize that I could simply instruct Kubuntu (or Ubuntu) to utilize the Canonical tracked binary drivers, however they’re always generally behind quiet a few versions, and for this system I like running the bleeding edge.

So my adventure in Nvidia driver installation begins yet again, a road I’ve been down many times…

Assuming you’ve downloaded the latest binary driver from Nvidia and try to install it you’ll immediately run into problems with the binary installer complaining that:

  • X windows is still running
  • The Nouveau module is loaded
  • The compiler can’t be found
  • 32bit libraries are missing (assuming you’re running 64bit OS like I am)

By far the most frustrating of these is Nouveau. Listen I’m all for open software, however the reality is that Nouveau is a poor substitute for the real deal from Nvidia. Would it be great if Nvidia open sourced their driver? Hell yes, however since they’re not willing to I’m stick using it in a binary format… Anyways, below are the steps I’ve used (recently) to achieve success in installing this driver:

  • Obtain the latest driver from Nvidia
  • Remove the packages xserver-xorg-video-nouveau and xserver-xorg-video-all
sudo dpkg -r xserver-xorg-video-nouveau xserver-xorg-video-all
  • Install the linux-headers package matching your kernel and the dkms package
sudo apt-get install linux-headers-`uname -r` dkms
  • Install GCC and make packages
sudo apt-get install gcc make
  • Install 32bit OpenGL libraries (if you’re running 64bit OS)
apt-get install libgl1-mesa-dri:i386 libgl1-mesa-glx:i386
  • Modify your GRUB configuration to set nomodeset on boot
  • edit: /etc/default/grub and find the line starting with: GRUB_CMDLINE_LINUX_DEFAULT. This usually this looks like:
  • Add nomodeset to it:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nomodeset"
  • Save the file and run update-grub
sudo update-grub
  • Reboot

Once rebooted make sure that X windows is shutdown. Usually if X is running on boot in Kubuntu or Ubuntu environments, it’s due to kdm, gdm, or lightdm

sudo stop lightdm

After that, try and run the nvidia binary driver installer and allow it to install the drivers. Hopefully all will go smoothly and your system will be ready to go. Once complete you can simply restart X windows:

sudo start lightdm

Good luck, and if you have problems feel free to let me know in the comments section!

Mounting a DVD or CD image virtually in Linux

Mounting a DVD or CD image (ISO file) virtually is trivial in linux. To do this you’ll need to have superuser permissions, in the example below I’m using the sudo command:

cfaber@sputnik:/dump$ mkdir ISO
cfaber@sputnik:/dump$ sudo mount -o loop DVD-IMAGE.ISO /dump/ISO
cfaber@sputnik:/dump$ df -h | grep loop
/dev/loop1             1.8G  1.8G     0 100% /dump/ISO

As we can see, mount was successful!

Use rsync to copy files faster, as well as showing progress

One of the commands that I find my self using almost everyday is rsync:

a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host
over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control
every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is
famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending
only the differences between the source files and the existing files in the destination. Rsync is widely
used for backups and mirroring and as an improved copy command for everyday use.

Rsync finds files that need to be transferred using a “quick check” algorithm (by default) that looks for
files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as
requested by options) are made on the destination file directly when the quick check indicates that the
file’s data does not need to be updated.

In general, the following flags work great: -P (Progress), -a (archive mode), -v (be a little verbose).   When copying to an NFS share, it’s also a good idea to bandwidth limit your connection to some reasonable number. I’m copying via wifi and I know from extensive testing that I get around 15 to 25 megabytes a second copies to my NFS server. So to stay on the safe side I limit to just under 15 megabytes (150000 bytes). This can be done with the –bwlimit=BYTES flag.


cfaber@sputnik:/home/dump$ rsync -Pav --bwlimit=15000 data /home/hal/OpenNAS/.
sending incremental file list
    608,370,688  56%   14.65MB/s    0:00:31 

And so on.

The nice thing about this is that rsync will respect that bandwidth limit. Additionally the file being synced is checksummed for errors, so you can be sure that the destination file is identical to the source file.

Rsync is also available on windows within Cygwin however, it doesn’t respect, and can’t copy file permissions (from within a windows environment).

Windows 7 SP1 and Samba

After successfully installing Windows 7 SP1 on my wife’s new workstation I noticed right away that the system was no longer able to access my Samba NAS. After searching around for a while and trying various tips, such as making sure the following Samba configuration values were enabled:

security = user

encrypt passwords = true

server signing = Auto

I was still having no luck. This was with Samba 3.6.6 and everything had worked prior to SP1 upgrade.

Looking around a bit more I discovered as of SP1, Microsoft enforces 128 bit encryption on file sharing and disables the older 40 – 56 bit encryption. This is great except it seems Samba doesn’t yet support this, at least not in the NT1 protocol mode (Which I’ve found to be the most stable).

So the fix was simple, under Advanced Sharing Settings, which is accessed from the left side panel in Network and Sharing Center you should find the option: Enable file sharing for devices that use 40- or 56-bit encryption


After changing that, I was able to once again access my Samba NAS without problems.

I guess I could have used security = ADS Samba configuration option, but I didn’t want to bother with the mess of active domain controls for a single windows box.

Another thing to look at is the LmCompatibilityLevel value in Windows registry. This key can usually be found in


If the REG_DWORD type key doesn’t exist, then you’ll need to create it with the name


Generally assigning a value of 1 is found to solve most people’s windows client issues.

Lastly, and this is critical, make sure you add the windows user to the samba user database, This can be done with the smbpasswd -a option. Neglecting to do so will cause you much frustration.

Unetbootin + Windows install

So… Unforantly for me, I had to reinstall Windows 7 on my wife’s work station. Personally I can’t stand Microsoft OS’s so this was quite the chore.  To make things easier I decided to just install via USB disk (Who uses CD’s anymore??). To do this I utilized my favorite application, unetbootin.

Sadly, this didn’t work. I neglected to have unetbootin install on an NTFS based USB stick.  After formating the stick to NTFS and remounting it (in read/write mode), I restarted unetbootin, this time with the installtype and targetdrive arguments:

unetbootin installtype=USB targetdrive=/dev/sdd1

Once started, I had it write out the image to the USB stick, popped  it into my wifes work station and was off to the races!