Monday, July 29, 2013

This blog is moving

While I will no longer be maintaining a presence here, I will leave the content in place for the foreseeable futrure.  Any updates as well as new posts will be maintained on my own domain -

Saturday, April 13, 2013

Use top in batch mode

Use top in batch mode

Assume a system is going down, and there's no explanation as to why, but perhaps oom_killer is being invoked. You might use top in batch mode to catch whatever process is causing a memory leak.

Whenever you run top, you can press Shift+F so that you can select the field by which to sort. We're looking for a memory leak so we would select 'N'. Note: 'N' is not a typo here:) Press enter to see each process, in order of the percentage of memory consumed. (You could also type lowercase 'f' to choose which fields are displayed.)

We'll want top, in batch mode, to open with this view the next time, and to do this, press Shift+W. This will save a file '.toprc' in the home directory. Now when top opens up it will have exactly the fields we wanted to be monitored, and the exact order. Back to batch mode!

top -b -n 1

Simple so far, b for batch mode; n for the number of iterations. I like to use just one, throw this into a script, and let cron handle the iterations. Like this:

date +'%Y%m%d %H:%M' >> /root/monitor.log
free -g >> /root/monitor.log
top -b -n 1 | head -n 12 | tail -n 5 >> /root/monitor.log
echo “-----” >> /root/monitor.log

crontab -e

Then you can specify the script to run every five minutes -
*/5 * * * * /root/bin/

In short, this will place a time stamp, followed by memory used in Gigabytes, followed by 5 of the processes using the most memory in /root/monitor.log.

Sunday, March 24, 2013

"Could not parse sgdisk output line: 1 64"

I ran into this earlier tonight while installing XenServer 6.0 on a whitebox. 

I'm not sure why this happens, but evidence strongly suggests it has something to do with the installer recognizing the partition table on the block device. (I couldn't say why this happens as the block device should be formatted anyway, but I digress.) 

This particular disk had been previously utilized in an mdadm array (partition type 0xFD).

If this happens, simply boot into a live environment, and delete any existing partitions.  In my case , I also recreated an array of a more common linux partition type 0x83 for good measure, and restarted the XenServer installation.

Monday, February 4, 2013

Sharing high resolution images with limited bandwidth

I ran into a problem about a year ago, when I first started taking pictures with a high megapixel camera. The original conception behind storing my pictures, was to have them hosted via a network service, like SMB or FTP. I found quickly that the network would in no way be quick enough to serve 24M pictures quick enough to be useful, and so access to family pictures and photographic pursuits remained bound to our workstation.

This last weekend, I finally got around to solving it. The solution is to offer a web page with low resolution images, each image being linked to a higher resolution original. This allows any client device in our house to get a quick preview of all pictures, as well as select one for a full resolution download.

This is easy enough to write up in html, but a home project like this never sees completion unless it's sufficiently scripted to save hours of repetitive typing.

I set up an FTP service to handle the full resolution images, and used apache to serve lower resolution ones. The script accesses my file structure which looks like this:


Most people who keep lots of content will have a method logical to themselves that could just as easily be parsed with a script, so the below should be fairly adoptable.

First I define a few locations to work with, including arguments so that the script doesn't need to be manually modified on every run.

YEAR=$1            # This is the first argument for the script
SUBJECT=$2       # This is the second argument
                       # (The script will be called with ./publish_photos year subject)
                       # These don't need to be defined, as $1 and $2 could be used
                       # at any time during the script, this just improves readability.
                       # PHOTO_LOC is the location of the full resolution picture
                       # currently mounted.
                       # Be sure that file permissions, Selinux contexts, (if you are
                       # using Selinux contexts), and firewall rules are set to allow
                       # access to both services.


                        # FTP_PHOTO_LOC is the same location, except as reached
                        # via url.


                        # HTML_DIR is the location that smaller images will be
                        # in. They will be served by Apache.


                        # The first step is to use ImageMagick's convert command
                        # to create our thumbnails, then move them to a folder hosted
                        # by Apache.

echo "Converting files."
for file in $( ls $PHOTO_LOC )
do convert $PHOTO_LOC/$file -resize 400x200 $PHOTO_LOC/ims-$file
echo "$file converted."
sleep 3
echo "Moving files."
mv $PHOTO_LOC/$HTML_DIR /var/www/html

                      # The next step is to create an actual page serve the
                      # pictures that are now in $PHOT_LOC/$HTML_DIR
                      # or /var/www/html/[year]_[subject]
                      # The script below will write a page that will serve all of the
                      # pictures. Because actual names of the pictures are the same
                      # except for an 'ims-' prefix, I can strip that part away to get
                      # the original name (See `echo $file | awk -F- '{print $2}'`).
                      # Three pictures will be printed before breaking for a new line.
                      # This behavior can be changed by raising from 3 to 4 the value
                      # in the statement `if (( $i % 3 == 0 )); then `
for file in $( ls /var/www/html/$HTML_DIR/ );
do echo "<a href=\"$FTP_PHOTO_LOC/$(echo $file | awk -F- '{print $2}')\"><img src=\"$HTML_DIR/$file\"></a>" >> /var/www/html/$HTML_DIR.html;
i=$(( $i +1 ))
if (( $i % 3 == 0 )); then
echo "<br>" >> /var/www/html/$HTML_DIR.html;

                      # This is just bare bones, and can be heavily modified to
                      # change the background of the serving web page, include
                      # CSS pages, or more practical improvements like sanitizing
                      # input.

The next step is easy enough that it would take more time to write a script than just banging out a quick landing page manually. You just need create an index.html file, and create links to each page hosting picures. To make it friendly to all of the touch devices that will inevitably fill your home in the coming years, I recommend making custom pictures as the links on the landing page as well.
Here's an example of my work flow going forward. I open the command line as root and type: 2012 20120325Sunrise

(20120325Sunrise is the $SUBJECT name as it's the folder name in my tree)

Given some time for the conversions to take place, this will populate my /var/www/html directory with a folder called 2012_20120325Sunrise, with low resolution images, as well as creating a webpage called 2012_20120325Sunrise.html.

I'll take one picutre from the set:

Modify with gimp:

I'll save the new file as sunrise_link_ims-DSC00733.jpg, and populate my demo landing page with:

<a href="2012_20120325Sunrise.html">
<img src="sunrise_link_ims-DSC00733.jpg">

Here's the result when I navigate to localhost using a web browser:

Once I click it the picture, it brings me to a page that looks like this:

The share page loads quickly, and each image provides access to full resolution versions with a single click.

Monday, January 21, 2013

Undo rm

When removing a symbolic link that points to a directory, there's a fine line between rm -rf /path/to/softlink and rm -rf /path/to/softlink/. (Note the trailing slash in the second example) And while quite easy to type the later, especially when using [TAB] to auto-complete, the actions taken by the system vary by a wide margin; the former will remove the pointer file that is the softlink, and the second will remove the actual source directory.

I had actually run into this problem earlier last night, and had the pleasure of watching literally tens of Gbs of pictures being removed from my filesystem, one by one.

$%@! -- CTRL-C! CTRL-C!”

From what was left, and all of my backups I figured I could recover all pictures except for 2009, 2010 and a 4 month gap in 2011 – not good enough.

If you find yourself in the same situation, there's a good program called PhotoRec (downloadable here) that will recover most deleted files – it works best for large chunks of data as it will scan an entire volume for a large variety of file types, outputting everything found to a chosen directory. Keeping this in mind it may not be worth the trouble for a 200 Kb text file. Here's the run down:

There's a downloadable version for most operating systems, and will work on most file systems. It wasn't in Fedora's repository that I could find, so I had to download directly and extract from a compressed file. The first thing that you'll notice is that the file downloaded will be called testdisk-6.13.linux24.tar.bz2 (this will vary depending on version) . This might be a bit confusing, as cgsecurity provides a program called testdisk as well – but this is in fact what you need as both programs are provided in the same download. Just extract wherever you want it, except of course the volume where data has been lost.

I didn't see a man page, or text files that explain all of the options, but luckily none are needed. Just navigate to the extracted directory (example: /path/to/testdisk-6.13) in your terminal and run ./photorec_static. It's a TUI (text-user interface), but you'll be able to navigate, and make selections easy enough with just the arrow keys. Choose the volume you want to scan, and the directory you want the recovered files placed.

Here's what it looks like in action:

 oooh, the excitement!

Whichever directory you chose will be filled with directories called recup_dir.##. PhotoRec fills them to varying degrees before creating another, incrementing the number, until done scanning the volume.

It's a bit messy, you'll have a lot to go through, and hopefully you've come to terms with your meta data being gone (no recognizable file names, original time of creation may be gone, etc.) but that's the nature of the beast.

Friday, January 4, 2013

Red Hat and iSCSI

One of the RHCE objectives is to “Configure a system as an iSCSI initiator that persistently mounts an iSCSI target.”

Doing so amounts to just a few commands. First, find and list available iSCSI targets using the iscsiadm command:

[root@localhost ~]# iscsiadm -m discovery -t st -p,1 iqn.2012-08.local.vmnet:10113-01

...where -m is for mode -t is for type, the st is for send targets. Finally, -p, specify the 'portal' followed by the ip address.

From the output I can see the iqn of a single lun, 'iqn.2012-08.local.vmnet:10113-01'.

Next, log in:
[root@localhost ~]# iscsiadm -m node -T iqn.2012-08.local.vmnet:10113-01 -p -l
Logging to [iface: default, target: iqn.2012-08.local.vmnet:10113-01, portal:,3260] (multiple)
Login to [iface: default, target: iqn.2012-08.local.vmnet:10113-01, portal:,3260] successful

The output indicates success, I could now mount your block device, which will should now be visible in /dev. If you're curious, you can log out using the same command, except followed by a -u as opposed to -l.

If you want this block device to automount, I would suggest adding the _netdev argument to your options in the /etc/fstab file. This will delay mounting of the block device until your network stack is up. This could prevent your system from hanging.

One other tip, if you have just a single lun being presented, you don't need the -T option at all, which could save you some typing. In effect, the same could be accomplished with just...

[root@localhost ~]# iscsiadm -m node -p -l

If you find yourself studying for the RHCE, you might find it useful to play with some of the commands yourself. Of course to do so, you'll need a set up a target. On the plus side, you can do this with Red Hat as well.

First you'll need to install the scsci-target-utils package and edit /etc/tgt/targets.conf. You'll want to place whatever storage device you'll offer as a 'backing-store' (or LUN) in between the <target> tags with the iqn name in the opening tag. As an example, here's my set up from the target referenced above:

<target iqn.2012-08.local.vmnet:10113-01>
backing-store /dev/vgiscsi/iscsiOne

Finally, ensure that the tgtd daemon is started (service tgtd start), and you should be good to go.

Thursday, December 20, 2012

RHEL 6 PXE install server from scratch v 0.2

Edit: This script will no longer be maintained here.  Updates can be found at

Per request I've updated the PXE server script.  Testing has been done with both 6.3 x64 as well as 6.2 i386, I don't have every revision to test with, but I see no reason why it wouldn't work with CentOS as well.

As always, read before you run.  Modification of the script header to your specifications is a requirement.

This script was originally developed as I was studying for the RHCE, and if you need an environment wherein machines could be quickly deployed, it works perfectly.  Once the PXE install server is deployed, simply place a script like the following in your bin directory:

if [ -z "$1" ]; then
echo "Please provide station number"
virt-install -n $STATION -r 512 --disk path=/var/lib/libvirt/images/$STATION,size=5 -w bridge=br0 --pxe --os-type=linux

Assuming you've named the file createStation, and your PXE server (created with the script below) is both set up and running, you would just need to type the following:

createStation 2

A virtual machine named station2.vmnet.local would be created.  You can see from above that it will be deployed with only 512 MB of RAM, and a 5 G hard drive, and very little intervention required.  Due to one of the updates, listed below, a PXE menu is now used for safety.  You will need to select your OS to install to keep it from attempting to boot to the hard drive.  This is necessary just in case something is accidentally booted to the network, it won't get it's hard drive wiped.

Additionally you can now choose via the $NETWORK_SHARE variable whether you would prefer an ftp server, or deploy via http, with the default being ftp.  That said the changes in full are listed below.


    - Added hostname variable
    - Host name is set when script is run
    - Added http option for PXE boot server
    - Clients that PXE boot will see menu, defaults to local hard drive for safety
    - openssh-clients in kickstart menu so you can immediate scp files to clients
    - changes selinux context in semanage database so changes survive a relabel
    - kickstart root password is now in plaintext, for easier modification of default start-up password.

...and below is the script that will build out your PXE server.  Just copy and paste:

# v.02
# You MUST edit variables to your environment
# or the deployment will be unsuccessful.
# READ script before executing!
# Note that the subnet variable is NOT your subnet MASK,
# rather the subnet (or network) on which the machine would reside.


# DHCP Range variables
# range dynamic-bootp

# Range

# PXE variable RHEL_V is a short descriptor of the Red Hat version
# This variable will be the name of a tftp directory that is created, so best to not include spaces

# Valid values for Network share as of this writing are:
# ftp | http

# Here is where the script will attempt check for errors
# before proceeding.
# If you would like to add additional protections before
# changes are made to the system, this would be the place
# to add them.

if [ ! -f $SOURCEPATH/media.repo ]; then
    echo "Please check that your RHEL CD is mounted at $SOURCEPATH"

[ $NETWORK_SHARE != "ftp" -a $NETWORK_SHARE != "http" ]; then
    echo "$NETWORK_SHARE is not an allowable value for \$NETWORK_SHARE, please revise and rerun the script."

cat > /etc/sysconfig/network << EOF


cat > /etc/sysconfig/network-scripts/ifcfg-$INTERFACE << EOF
HWADDR="$(cat /etc/udev/rules.d/70-persistent-net.rules | grep $INTERFACE | awk -F\" '{print $8}' | tr [a-z] [A-Z])"

cp /media/media.repo /etc/yum.repos.d/
cat >> /etc/yum.repos.d/media.repo << EOF
yum clean metadata

# Install services
yum -y install dhcp tftp-server syslinux policycoreutils-python

# Adjust firewall
cat > /etc/sysconfig/iptables << EOF
# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 21 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 53 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 53 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 69 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 69 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
service iptables restart

# Configuration of chosen service and Kickstart file
cat > /root/ks.cfg << EOFF
# Kickstart file automatically generated by anaconda.

url --url $NETWORK_SHARE://$IPADDR/inst
lang en_US.UTF-8
keyboard us
network --onboot yes --device eth0 --bootproto dhcp
rootpw RedHat!
firewall --service=ssh
authconfig --enableshadow --passalgo=sha512
selinux --enforcing
timezone --utc America/North_Dakota/Center
bootloader --location=mbr --driveorder=vda --append=" rhgb crashkernel=auto quiet"
# The following is the partition information you requested
# Note that any partitions you deleted are not expressed
# here so unless you clear all partitions first, this is
# not guaranteed to work
part /boot --fstype=ext4 --size=500
part pv.253002 --grow --size=1

clearpart --all --drives=sda
volgroup VolGroup --pesize=4096 pv.253002
logvol / --fstype=ext4 --name=lv_root --vgname=VolGroup --grow --size=1024 --maxsize=51200
logvol swap --name=lv_swap --vgname=VolGroup --grow --size=512 --maxsize=1024

#repo --name="Red Hat Enterprise Linux" --baseurl=cdrom:sr0 --cost=100

%packages --nobase
cat >/etc/yum.repos.d/media.repo <<EOF
name=Red Hat Server Linux 6

# DHCP configuration
cat > /etc/dhcp/dhcpd.conf << EOF
# DHCP Server Configuration file.
# see /usr/share/doc/dhcp*/dhcpd.conf.sample
# see 'man 5 dhcpd.conf'
subnet $SUBNET netmask $NETMASK {
range dynamic-bootp $BEGIN1 $END1 ;
range $BEGIN2 $END2 ;
option routers $IPADDR;
default-lease-time 6000;
max-lease-time 7200;
allow booting;
allow bootp;
class "pxeclients" {
match if substring(option vendor-class-identifier, 0,9) = "PXEClient";
next-server $IPADDR;
filename "pxelinux.0";

# TFTP server configuration
mkdir -p /var/lib/tftpboot/$RHEL_V
mkdir -p /var/lib/tftpboot/pxelinux.cfg
cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/
cp /usr/share/syslinux/menu.c32 /var/lib/tftpboot/

# pxe default file configuration
cat > /var/lib/tftpboot/pxelinux.cfg/default << EOF
default menu.c32
prompt 0
timeout 150


label local
MENU LABEL Boot to hard drive

label $RHEL_V
kernel $RHEL_V/vmlinuz
append initrd=$RHEL_V/initrd.img noipv6 ks=$NETWORK_SHARE://$IPADDR/pub/ks.cfg ksdevice=$INTERFACE


    yum -y install vsftpd
    mv /root/ks.cfg /var/ftp/pub/ks.cfg
    chmod 755 /var/ftp/pub/ks.cfg
    restorecon -R -v /var/ftp/pub
    mkdir -p /var/ftp/inst
    cp -var /media/. /var/ftp/inst/
    semanage fcontext -a -t public_conent_t "/var/ftp/inst(/.*)?"
    restorecon -R -v /var/ftp/inst/

    iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 21 -j ACCEPT
    service iptables save

    service vsftpd start
    chkconfig vsftpd on

    # Add appropriate line to modules to /etc/sysconfig/iptables-config
    sed -i 's/IPTABLES_MODULES=""/IPTABLES_MODULES="ip_conntrack_ftp ip_conntrack_tftp"/' /etc/sysconfig/iptables-config

    # Readjust repo
    sed -i 's/baseurl=file:\/\/\/media\/Server/baseusr=file:\/\/\/var\/ftp\/inst\/Server/' /etc/yum.repos.d/media.repo
    yum clean metadata
    # Finish PXE configuration
    cp /var/ftp/inst/images/pxeboot/* /var/lib/tftpboot/$RHEL_V/

    yum -y install httpd
    sed -i 's/#ServerName $SYSNAME$DOMAINNAME/g' /etc/httpd/conf/httpd.conf

    mkdir /var/www/html/pub
    mv /root/ks.cfg /var/www/html/pub/ks.cfg
    chmod 755 /var/www/html/pub/ks.cfg
    semanage fcontext -a -t httpd_sys_content_t "/var/www/html/pub(/.*)?"
    restorecon -R -v /var/www/html/pub
    mkdir /var/www/html/inst
    cp -var /media/. /var/www/html/inst/
    semanage fcontext -a -t httpd_sys_content_t "/var/www/html/inst(/.*)?"
    restorecon -R -v /var/www/html/inst

    iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
    service iptables save

    service httpd start
    chkconfig httpd on

    # Readjust repo
    sed -i 's/baseurl=file:\/\/\/media\/Server/baseusr=file:\/\/\/var\/www\/html\/inst\/Server/' /etc/yum.repos.d/media.repo
    yum clean metadata
    # Finish PXE configuration
    cp /var/www/html/inst/images/pxeboot/* /var/lib/tftpboot/$RHEL_V/

# Start services, ensure services start with system boot
service xinetd start
chkconfig xinetd on
chkconfig tftp on
service dhcpd start
chkconfig dhcpd on
service network restart
service iptables restart
echo ""
echo "Set up is complete.  You are now free to test"

# Roger Heslop
# 20121220