HowTo

Dell PowerEdge 13th Gen Fan Noise

I recently came across the opportunity to assist a client with installing their new Dell PowerEdge R730XD. Quite the beefy server config, 2x10core CPUs, 128GB of RAM, 12x4TB NL-SAS, you know, all the goodies. This machine is slated to replace an aging T610 that has seen better days performance-wise.

I went ahead and put an Intel 10GbE card in the server since all other hosts in the server room including both backup boxes are 10GbE enabled and are connected to our new Netgear 10GbE switch. Keep in mind this was an industry standard PCIe 10GbE card, a particularly good one, the Intel TX540-2. After installing VMware ESXi, and later, Windows Server 2012 R2, users were complaining about the loud “jet sounding” noise coming from the server room. After logging into the Dell iDRAC Enterprise card I immediately noticed that the fans were running around 92% which was roughly 15K RPM or thereabouts. This was regardless of operating system mind you, so I couldn’t even blame Windows OR VMware this time.

After looking around online at various forums I realized that the system was running the fans near max speed/volume due to the presence of a non-certified PCIe card installed into the system. For all intents and purposes, non-certified means you didn’t pay through the nose to acquire the identical hardware from Dell. Essentially, since the Intel card doesn’t carry the Dell specific code/firmware to report back that “all is well over here in PCIe/temperature land”, the system defaults to running the fans in jet engine mode. For posterity’s sake and to clarify, this will happen on pretty much any non-Dell card that is inserted. In researching the issue I found numerous folks that put actively cooled GPUs, old school 4x1Gbps network cards, you name it, high speed fan noise.

Well no big deal, all you have to do is go into the Dell BIOS and modify a setting or two so that the system doesn’t run the fans at full steam when a card inserted right? Wrong! That would be the logical assumption and design choice to make so you know they didn’t make it that easy. Read on below to understand how I finally got this system to quiet down. The info below is compiled from many sources and some of my own figuring out, just though it would be helpful to have it all in one place.

Step 1: Enable IPMI
For this step enter your Dell servers setup/config screen and get to the remote access configuration/iDRAC setup. In the iDRAC setup you need to do all of the standard stuff like assigning an IP and setting user credentials etc, but you MUST also turn set “Enable IPMI over LAN” to yes. This setting is crucial to completing the steps below successfully.

Step 2: Get IPMI tools
Linux users can use their preferred package/distribution method to obtain ipmitool while Windows users will need to grab the Dell OpenManage BMC Utility and get it installed.

Next, open up and command prompt and navigate to the directory the BMC utility installed to, on my system this was: C:\Program Files (x86)\Dell\SysMgt\bmc\

From there go you will see several files, the program that we are using here is ipmitool.exe. Go ahead and run ipmitools.exe without any switches/arguments just to make sure its installed and working.

Step 3:
The third and final step is essentially ‘the fix’. This is where you can check the status, and then disable or enable the systems cooling response to third party cards that are installed on the PCIe bus. This part was a little frustrating at first because I was working in the right direction and was just about there but the commands weren’t being sent or interpreted the way the should have been.

You must use the lanplus option instead of lan but it is important to note that lanplus does NOT work unless you’ve enabled the “Enable IPMI over LAN” setting that I mentioned back in step 1. The non-intuitive part about that was that although I was running the right command aside from lan vs lanplus, I really didn’t get any clear feedback as to why the command wouldn’t “take”.

Anyhow, here is the base command which you need to acquaint yourself with:

ipmitool -I lanplus -H ipaddress -U root -P password raw

Obviously you will need to substitute your own iDRAC ip, user, and password. After that, just tack on one of the three commands below.

Disable Third-Party PCIe Card Default Cooling Response:
ipmitool -I lanplus -H ipaddress -U root -P password raw 0x30 0xce 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x01 0x00 0x00

Enable Third-Party PCIe Card Default Cooling Response:
ipmitool -I lanplus -H ipaddress -U root -P password raw 0x30 0xce 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x00 0x00 0x00

To check the current third party PCIe card default cooling setting:
ipmitool -I lanplus -H ipaddress -U root -P password raw 0x30 0xce 0x01 0x16 0x05 0x00 0x00 0x00

This response means disabed:
16 05 00 00 00 05 00 01 00 00

This response means enabled:
16 05 00 00 00 05 00 00 00 00

After disabling the third party cooling response my system went from the previously mentioned 15K RPM mark down to a user verified sane noise level/speed of around 6K RPM.

A key takeaway and disappointment for me is that in this day and age of widespread standards and simplicity, things are becoming increasingly proprietary and complex.

–himuraken

Got old-buntu? Ubuntu EOL 9.10 to 10.04 Upgrade Mini HowTo

So several months ago, I like the rest of the world, was notified that end of life (EOL) for Ubuntu 9.10 Karmic Koala would happening. In the news blurb/mailing list, wherever I found it, I walked away thinking that security updates would cease to exist.

In preparation for the upgrade, I went ahead and cloned the 9.10 server and proceeded to upgrade the server to Ubuntu 10.04 Lucid Lynx. This went off without a hitch from what I could tell and I scheduled the upgrade of the production server with my last client running 9.10.

Without fail, life happens, clients have things come up, and the upgrade never happened. Fast forward to present day and time, and my client tried installing a package using apt-get and received a slew of errors. Looking into the issue a bit further and I found the repositories gone. Interestingly enough, when EOL occurs for an Ubuntu release, it really ends, and not just for the security patches.

So one is left wondering, “how can I sudo apt-get install update-manager-core & sudo do-release-upgrade when I can’t even do a simple sudo apt-get update?” Solution: EOL upgrade. There are several different ways to go about this, the best are detailed here. At the time of this writing, the link is a little unclear about how to get 9.10 to 10.04 so here is the quick and easy way:

1. Backup your current sources.list:
sudo mv /etc/apt/sources.list ~/sources.list

2. Create a new sources.list:
sudo vim /etc/apt/sources.list

3. Add/paste in archive release repositories substituting CODENAME for release jaunty, karmic, etc:

## EOL upgrade sources.list
# Required
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-updates main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-security main restricted universe multiverse

# Optional
#deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-backports main restricted universe multiverse

4. Update repositories and install update manager
sudo apt-get update
sudo apt-get install update-manager-core

5. Initiate the upgrade
sudo do-release-upgrade

6. Enjoy!

–Himuraken

Convert Thick Virtual Disks to Thin

When working with virtual machines, it is often advantageous to over allocate and under utilize resources. When it comes to virtual hard disks, this is even more common place. On low use or low demand servers, I always use thin provisioning. This saves disk space by only using physical disk space when the guest actually uses the virtual disk. But what about those disks that were created using the thick option, or brought over as thick automatically during a P2V conversion? Time to convert your thick virtual disk to thin.

As always, I recommend backing up all of your data and knowing what you are doing. Test this in a non-production environment.

Converting disks from thick to thin is actually quite easy and can be accomplished using these steps:

1. Log into your ESX host using SSH and cd into the VM directory that contains your virtual disk.

2. Shutdown the VM so that we can get exlusive access to the virtual disk.

3. Run vmkfstools -i yourthickdisk.vmdk -d thin yourthindisk.vmdk

4. Edit the settings for your VM and remove the existing drive. Add a new hard drive and choose the existing drive option.

5. Boot the VM and enjoy.

Note: Dont forget to go back to ESX server and remove the old .vmdk and -flat.vmdk files once you are sure that your VM is operating normally off the thin disk.

–Himuraken

GDesklets won’t run in Ubuntu 7.10 Gutsy Gibbon

After a clean install/upgrade of Ubuntu 7.10 (Gutsy Gibbon) you may be unable to get gdesklets to run. I have noticed several threads out there with no resolution so I thought I would post a fix. Most users will get the app installed successfully but when they run the gdesklets shell, the app begins to load, goes gray, and then has to be closed using force quit. Also, running gdesklets start from a console shows that the daemon is starting but it never does. By default Gutsy Gibbon uses Python 2.5 while gdesklets is looking for 2.4. This is pretty straight forward to resolve using these steps. Install python2.4 by running sudo apt-get install python2.4 from a console. After that is complete we need to tell gdesklets to use Python2.4. From the console we need to add a few entries to the gdesklets config files. Locate the line that says “#! /usr/bin/env python” and append it with a 2.4. So each line should now look like “#! /usr/bin/env python2.4”. This line needs to be updated at the top of each of the files listed below.

/usr/lib/gdesklets/gdesklets
/usr/lib/gdesklets/gdesklets-shell
/usr/lib/gdesklets/gdesklets-daemon
/usr/lib/gdesklets/gdesklets-migration-tool

After updating the files with your favorite text editor, start the app by running gdesklets start from a console. This should have you up and running in no time.

–himuraken