thoughts from two guys

So I used to use virtual PXE booted floppies to handle various tools that my cluster nodes required to update firmware and other misc items that often come as DOS based programs. This worked well, but I got lazy and didn't track the fine features of MEMDISK from the SYSLINUX closely enough when most of these tools started showing up on ISO images.

So I missed that they recently added ISO booting support to MEMDISK. And thanks to a very well written article I've moved a pile of my common "utility" ISO images into a PXE bootable menu. (And not all of these have to do with clusters below)

So here is what I did:

• Go get the latest version of Syslinux just to be sure
  

• Place the latest pxelinux.bin into your tftp server which you probably already have and add "memdisk" to the same location

• Also copy the menu.c32 file. that provides a basic menu system for PXE boots

Then, in your pxelinux.cfg area I've made a file called "chooser" which when I want to boot a system off that config, I symlink the needed IP address file to it. It contains right now:


UI menu.c32
DEFAULT bios

LABEL bios
MENU LABEL FreeDOS BIOS Tool ISO
KERNEL memdisk
APPEND iso initrd=isos/bios.iso

LABEL ntchpass
MENU LABEL NT Change Password ISO
KERNEL memdisk
APPEND iso initrd=isos/ntchpass.iso

LABEL bartpe
MENU LABEL Bart PE ISO
KERNEL memdisk
APPEND iso initrd=isos/bartpe.iso

LABEL diskzapr
MENU LABEL Diskzapr
KERNEL memdisk
APPEND iso initrd=isos/diskzapr.iso


LABEL seagate
MENU LABEL Seagate Firmware Disk ES ST31000340NS Series SN06
KERNEL memdisk
APPEND iso initrd=isos/ES2SN06B-GOCHECK.iso



The first one is where I've been collecting various BIOS flashing tools for quite some time. Its a FreeDOS based disk with just a set of directories sorted by vendor and board name for the various tools. It was better than trying to mash all the tools together to have the room of the ISO image.

That last one, DO NOT just blindly deploy Seagate Firmware updates for the recent problems without fully reading the Seagate KB article on which ones are right for your drive. I've changed the ISO name to prevent that.

So I've spent a few weeks catching up on the latest in a slightly different type of "cluster": Highly Available clusters via the ongoing project formerly known as Heartbeat.

Pacemaker

I was able to get some of the basic stuff going quite easily under the new CRM system (compared to the very easy to use Heartbeat config file of the good old days) and even some basic Libvirt redundant KVM based system moving around. Not live migration of active network connections, but rapid relocation of a KVM node based on a shared backend storage.

The goal for this is a paired set of redundant servers that provide "utility" functions to a cluster. Be it an Kickstart install server, a Ganglia server, etc.

Good gravy, its 2010 already. We ended up being severely overloaded following SC09 and between the two of us have been tuning and working on two new cluster projects. We had a great time at SC09 and met a fantastic group of people. We will try to summarize some of the new things we've come across in the next few days! Happy New Year!

As I've worked on this project I've fired some ideas back and forth with the very savvy crew at Flad.com in a SILC chat room. They've been stunningly helpful and donated to me some spare equipment and their electrical skills as well. Some subtle issues would not have been solved without their ideas.

Heck, they even came up with a good box to ship this cluster to SC09.

So here's a very thankful shout to: evolo, celt, syntaxian, and lemonlime.

Beer's on me next outing.

OK, spent some time on the software side of the fence for demo/training purposes. After looking at ROCKS for awhile, I decided I liked to "roll my own" environment a bit more than use their methods (which are fine, but I'm pretty set in my ways).

I've got the four nodes now configured with

• Sun Gridengine
• Lustre (very basic single OST config for now)
• Cfengine (still version 2, working on 3)
• lm_sensors with some tweaks to read from the Intel boards
• Ganglia, Cacti, Nagios

Some highlights have been getting lm_sensors updates from the fine folks at Elrepo and having the w83627dhg module start working.

mcr1u1 [~]> sensors
w83627dhg-isa-0a00
Adapter: ISA adapter
VCore:     +1.03 V  (min =  +0.00 V, max =  +1.74 V) 
12VCC:    +10.56 V  (min = +13.04 V, max =  +0.74 V) ALARM
AVCC:      +3.42 V  (min =  +0.85 V, max =  +2.66 V) ALARM
3VCC:      +3.42 V  (min =  +2.43 V, max =  +3.97 V) 
in4:       +1.86 V  (min =  +0.94 V, max =  +0.15 V) ALARM
in5:       +1.22 V  (min =  +0.11 V, max =  +1.08 V) ALARM
5VCC:      +5.12 V  (min =  +6.43 V, max =  +4.38 V) ALARM
VSB:       +3.44 V  (min =  +1.54 V, max =  +1.76 V) ALARM
VBAT:      +3.31 V  (min =  +1.81 V, max =  +0.42 V) ALARM
Case_Fan:    0 RPM  (min = 3515 RPM, div = 32) ALARM
CPU_Fan:     0 RPM  (min = 2109 RPM, div = 128) ALARM
Aux_Fan:     0 RPM  (min = 2812 RPM, div = 32) ALARM
fan4:        0 RPM  (min = 5625 RPM, div = 16) ALARM
fan5:        0 RPM  (min = 84375 RPM, div = 16) ALARM
Sys_Temp:    +35 C  (high =   +18 C, hyst =   -96 C)  [CPU diode ] ALARM
CPU_Temp:  +47.5 C  (high = +80.0 C, hyst = +75.0 C)  [CPU diode ]
AUX_Temp: +127.0 C  (high = +80.0 C, hyst = +75.0 C)  [thermistor] ALARM
vid:      +2.050 V

My initial reaction was "Ahh! The Fans are not spinning!" but then I calmed down remembering the board is fanless . I also need to look at the config for the module and compare to the BIOS values for the voltages. They seem a little off.

The head node uses a f71882fg hwmon driver and after reading some MSI docs, I re-labelled the outputs to what I think are correct values. I've attached the power supply fan to one of the board fan pins

[root@mini etc]# sensors
f71882fg-isa-0a10
Adapter: ISA adapter
3VCC:      +3.36 V
Vcore:     +1.18 V  (max =  +2.04 V)  
Vdimm:     +4.08 V
Vchip:     +4.08 V
5VCC:      +5.04 V
12VCC:    +14.27 V
5VSB:     +10.71 V
3VSB:      +3.36 V
Battery:   +3.36 V
PSU_Fan:  1043 RPM  
CPU_Fan:  5660 RPM  
CPU_Temp:    +41 C  (high =   +85 C, hyst =   +81 C)  
                    (crit =  +100 C, hyst =   +96 C)  sensor = transistor

On the cosmetic side, I've gotten some proper length SATA cables with 90 degree bends in them.

Hey, and UPS delivered another 2GB stick of memory for the head node, bringing the total "cluster" to 10G of ram total. (2GB already in the three nodes, but I forgot I put a 1GB stick in the head node).