Perl link extractor

Below is a simple script I use almost daily which I thought I would share with you all. The script fetches various URL’s then extracts all hyper links from the fetched data. It cleans them up a bit and prints the resulting data as standard output

#!/usr/bin/perl
use strict;
use HTML::LinkExtractor;
use LWP::Simple;

if(!$ARGV[0]){
        print "Usage:
        $0 URL URL URL ...
";
exit;
}


# Fetch and parse the link
for my $link (@ARGV){
        my $LX = new HTML::LinkExtractor;
        my $page = get($link);
        $LX->parse(\$page);

        # figure out URL base.
        my $base;
        if($link =~ /^(https?:\/{2}[^\/]+)\/?/i){
                $base = $1;
        }

        if($link !~ /\/$/){
                my @link = split(/\//, $link);
                pop @link;
                $link = join('/', @link);
        }

        for(@{ $LX->links } ){
                if(lc $_->{tag} eq 'a'){
                        my $url;
                        if($_->{href} =~ /^\//){
                                $url = $base . $_->{href};
                        } else {
                                $url = $link . '/' . $_->{href};
                        }

                        print qq{$url\n};
                }
        }
}

I call it linkext. Below is an example usage to fetch all of the links available on the Intel Lustre download page:

$ linkext https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64/
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//?C=N;O=D
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//?C=M;O=A
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//?C=S;O=A
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//?C=D;O=A
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-2.6.32-431.20.3.el6_lustre.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-debuginfo-2.6.32-431.20.3.el6_lustre.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-debuginfo-common-x86_64-2.6.32-431.20.3.el6_lustre.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-devel-2.6.32-431.20.3.el6_lustre.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-firmware-2.6.32-431.20.3.el6_lustre.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-headers-2.6.32-431.20.3.el6_lustre.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-debuginfo-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-dkms-2.6.0-1.el6.noarch.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-iokit-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-modules-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-osd-ldiskfs-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-osd-zfs-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-source-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-tests-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//perf-2.6.32-431.20.3.el6_lustre.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//perf-debuginfo-2.6.32-431.20.3.el6_lustre.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//python-perf-2.6.32-431.20.3.el6_lustre.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//python-perf-debuginfo-2.6.32-431.20.3.el6_lustre.x86_64.rpm
https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//sha256sum

So then, to have this a bit more useful lets parse it with egrep and pass the arguments to xargs, which executes wget to fetch our files:

linkext https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64/ | egrep "\.rpm$" | xargs wget

Which would start downloading the various files. Or you can of course have xargs execute echo and display the full command line in which wget is to work on:

$ linkext https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64/ | egrep "\.rpm$" | xargs echo wget
wget https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-2.6.32-431.20.3.el6_lustre.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-debuginfo-2.6.32-431.20.3.el6_lustre.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-debuginfo-common-x86_64-2.6.32-431.20.3.el6_lustre.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-devel-2.6.32-431.20.3.el6_lustre.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-firmware-2.6.32-431.20.3.el6_lustre.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//kernel-headers-2.6.32-431.20.3.el6_lustre.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-debuginfo-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-dkms-2.6.0-1.el6.noarch.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-iokit-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-modules-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-osd-ldiskfs-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-osd-zfs-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-source-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//lustre-tests-2.6.0-2.6.32_431.20.3.el6_lustre.x86_64.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//perf-2.6.32-431.20.3.el6_lustre.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//perf-debuginfo-2.6.32-431.20.3.el6_lustre.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//python-perf-2.6.32-431.20.3.el6_lustre.x86_64.rpm https://downloads.hpdd.intel.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64//python-perf-debuginfo-2.6.32-431.20.3.el6_lustre.x86_64.rpm

I hope you all find this useful.

Parse smb.conf based on an NFS mapping directory

So I’m not sure how others run their home networks but in my case I have a single large NFS NAS. This NAS hosts both Samba and NFS shares as well as a web portal.

After many years of trying different configurations to my primary data stores I’ve finally come up with a simple method which is shared across all clients. The way this is done is to export a directory called ‘mapping’, in which it maps back to the mounts and sub directories containing my various Music, Pictures, Software, categories. These are mapped via symbolic links. An example of this would be:

Music -> /nfs/hal/data1/Music

Well this works great, as all of the clients accessing the data (including the web portal) all mount my various arrays (data0 – data4) this way. This also works out well if I have problems and want to shift data around, I can quickly remove the Music link and say point it to /nfs/hal/data4/Music.

Now onto the Samba part, even though samba seemingly has zero problems following symbolic links, you can’t point a share directly to one. I have no idea why and really don’t want to try and get into the mess that is samba debugging. So the quick solution there for me is to generate a smb.conf template file. It’s basically my full blown smb.conf file except all path references are replaced with place holders. I.e.:

[Music]
comment = Music
writable = no
browseable = yes
guest ok = yes
create mask = 0777
path = %Music%

This why I can automate the replacement of paths based on my mapping directory. Below is the simple script I’m using to do this:

#!/usr/bin/perl

use strict;

our $mapping = '/exports/mapping';
our $template = '/etc/samba/smb.conf.template';

my %list;
if(opendir(my $dh, $mapping)){
        for(readdir $dh){
                if($_ eq '.' || $_ eq '..'){
                        next;
                } else {
                        my $path = readlink($mapping . '/' . $_);
                        if($path){
                                $list{ $_ } = $path;
                        }
                }
        }

        closedir($dh);
} else {
        die "Unable to read directory: $mapping $!\n";
}

if(open(my $fh, $template)){
        my $data;
        { local $/; $data = <$fh>; };

        for(keys %list){
                $data =~ s/\%$_\%/$list{$_}/g;
        }

        print $data;
}

exit;

Simple script to alert you on ZFS pool problems

So after looking around this morning for a simple file system script to alert me when my ZFS pools have problems on my new file server I found a few, the problem was none worked correctly. Strangely these scripts were extremely simple so I was a bit surprised any hand problems. Because of this I took five minutes to knock together a portable perl script to email you in the event of problems:

#!/usr/bin/perl

our @users = qw(root);
our $zpool = "/sbin/zpool";
use strict;
use Sys::Hostname;
use IPC::Open3;

our $stat;
chomp($stat=`zpool status -x | head -1 | awk '{print \$4}'`);

if($stat ne 'healthy'){
 my $data = ("Date: " . scalar(localtime()) . "\n") .
 ("Hostname: " . hostname()) .
 "\n" .
 `$zpool status -x` .
 "\n";

 my $pid = open3(\*CHLDIN, \*CHLDOUT, \*CHLDERR,
 "mail", "-s", "ZFS Pool errors detected: " . hostname(), @users);

 print CHLDIN $data;
}

Hacking the IBM M1015 a.k.a. LSI 9640-8i RAID controller

As of late I’ve been diligently working on a new file server system for my home network. The current file server I have utilizes an old Supermicro AOC-USASLP-L8i RAID card. This worked well to handle the many drives I have, however now It’s time to upgrade, and while I’m at it, expand the overall capabilities of the system. So I picked up a set of sub-$100 LSI 9640-8i RAID controllers from eBay. After a quick search I found a very nice write up from a guy going by the handle kkm on the Lime Technologies forums. The write up was great, however I did learn a few things which might help the next person out, so I’m going to repost it here with a few modifications and notes on what I experienced.

GOAL

Replace the current M1015 firmware with an IT (HBA pass through) so as to utilize the controller as a JBOD driver.

STEP 1 Generate your boot disk

Create a bootable FreeDOS USB flash drive. This can be done easily with the Unetbootin utility.

STEP 2 Grab the files you need

Download the m1015 IT flash utilities from here or from the original location on Mediafire. Once downloaded, unpack the archive onto the root of your newly created FreeDOS bootable USB flash drive

STEP 3 Select a system and install the LSI card

Prior to installing the card in your system you’ll want to grab the SAS address of the card. This can be found on the back side of the card, directly below the PCB cut-away. This address always starts with 500605b:

IMG_20140815_114138

In the above example the complete SAS address will be: 500605b004e9d20c

Be sure to write this down, you’ll need it later!

Next, for this process to work you must utilize a system which supports at least PCI express 4x or better. You will also need to pick a system which doesn’t currently utilize an LSI controller or LSI variant. Unfortunately many Intel based systems won’t work because of problems associated with the sas2flsh utility from LSI. I found this to be true when attempting this procedure on my new LGA 1155 based board. The quick and dirty solution was to pull out my old AM2 based system and use that. For those interested in replicating this system, it was an ASUS M2A-74AM motherboard with some AM2+ chip and ram. Really the motherboard is the most critical part of this process.

Please also note again that this process will only be successful when flashing 1 LSI card at a time!

STEP 4 Boot FreeDOS

Boot up your newly created FreeDOS USB Flash drive, when prompted by the FreeDOS menu, select option (5) FreeDOS Live CD only. I’ve found this to be the most stable as other options tried resulted in instant crashes of FreeDOS with this set of tools.

Also remember, this is DOS, all of your files (which you extracted to the root of USB Flash drive) will be available under the C: drive.

STEP 5 Backup your current Serial Boot ROM

To restore your card to factory defaults, at some point in the future, or the back out of the changes you’re making now, it’s a good idea to create a backup of your Serial Boot ROM (SBR). To do this run the following command:

megarec -readsbr 0 <backup file name>.sbr

Where <backup file name> is replace it with the name you wish to save the SBR file as. (I simply used: backup)

STEP 6 Clear the Serial Boot ROM and erase the controller firmware

To clear the Serial Boot ROM and erase the controller, first write the empty.bin file back to the controllers Serial Boot ROM, then erase the firmware:

Clear the SBR with the following command:

megarec -writesbr 0 empty.bin

Now erase the controller flash memory. (This procedure is safe and recoverable)

megarec -cleanflash 0

Once complete, power cycle your system. A simple warm boot may do the tick however it’s recommended that you completely power down and then back up your system prior to the next steps.

Also note that upon reboot you will no longer see the LSI boot menu and notices, as these have been erased.

STEP 6 Writing the new firmware and re-adding the SAS address

The final step in this process is to write the new firmware back to the controller and then reassign the SAS address which we obtained earlier from the back of the controller card (see above)

Using this command, flash the provided 2108it.bin firmware along with the mtpsas2.rom:

sas2flsh -o -f 2108it.bin -b mptsas2.rom

At this point your controller should flash with the new firmware image, however if you received an error such as “Failed to initialize PAL…” then you’ll need to try this procedure again with a different motherboard.

Assuming your flash was successful, next re-address your controller card with it’s SAS address:

sas2flsh -o -sasadd <address>

In my case, address would be: 500605b004e9d20c

And that’s it! If all went well on your next reboot, your card should come online as a simple HBA pass through!

 

Restoring the controller back to factory defaults

To restore the controller back to factory defaults obtain the factory firmware from LSI’s website, extract the firmware file imr_fw.rom to your USB flash drive produced in step 1, and follow steps 2, 3, 4, and 6. Once complete write your backup Serial Boot ROM back to the controller:

megarec -writesbr 0 backup.sbr

Or what ever name you’ve chosen for your backup file. Once complete write the factory flash rom image back to the controller:

megarec -m0flash 0 imr_fw.rom

After that finishes, power cycle your system. Please note that upon reboot, your controller will appear to hang, this is nothing to worry about as the hang is the controller itself reconfiguring for the new firmware. This takes anywhere from 3 to 5 minutes so just be patient.

Finally finished with FP836 restoration

Well I finally had some time at the end of the day today to finish the FP836 front panel restoration. About the only thing it needed was two new LED’s for the faults. Unfortunately to my dismay the thermal fault LED traces on the PCB appeared bad, but lucky for me super micro drilled a convenient hole for me and I happened to have some 30g wire laying around. It’s clearly a bodge but hey it works and you don’t even notice it once it’s deep within the case!

IMG_20140811_174311

Pinout for Supermicro FP836 front panel connector

Recently I purchased a used SC836 Supermicro chassis, it was pretty much stripped to the bare bones. My plan is to make this into a file server, however I don’t want to utilize a Supermicro motherboard. Lucky for me it accepted the mini-ATX board I had, however the front panel connector utilized a funky 16pin IDC connector, unlucky for me the cable supplied isn’t long enough so I’ll have to sort that out later. Anyways, after some quick probing heres the pinout for this connector (since I can’t seem to find one online)

 

16 PIN IDC with Odd pins being positive and Even pins being Negative:

+ [01] Pwr Sw+ [03] Rst Sw+  [05] Pwr Fault+  [07] Thermal Fault+  [09] Net 2+  [11] Net 1+  [13] Hdd+  [15] Pwr+
- [02] Pwr Sw- [04] Rst Sw-  [06] Pwr Fault-  [08] Thermal Fault-  [10] Net 2-  [12] Net 1-  [14] Hdd-  [16] Pwr-

OR:

01 Power Switch+
02 Power Switch-
03 Reset Switch+
04 Reset Switch-
05 Power Supply Fault LED+
06 Power Supply Fault LED-
07 Thermal Fault LED+
08 Thermal Fault LED-
09 Network Interface LED 1+
10 Network Interface LED 1-
11 Network Interface LED 2+
12 Network Interface LED 2-
13 Hard Disk Drive Activity LED+
14 Hard Disk Drive Activity LED-
15 Power On LED+
16 Power On LED-

Hope this helps someone.