Righteous Hack

For a myriad of reasons, I have been looking at alternatives to VMware ESXi for a few months. Virtualizing a few machines here and there has proven educational. Learning the ropes of working with qemu/kvm, libvirt, and virsh has been challenging at times, but overall a pleasure to work with. Working with kvm is great although it takes some getting use to coming from a VMware/ESXi centric environment.

Up to this point all of the virtual machines that I had worked with were new systems. After some research and a few backups of my current VMs running on one of my ESXi hosts, I decided to migrate a few production VMs. Here are the steps that I used to move virtual machines over from a licensed vSphere 4.1 installation to a Linux host running qemu/kvm.

For starters, be sure that you have full backups of any VMs that you plan on working with. With that out of the way, you are ready to start:

1. Remove all snapshots from the virtual machine across all virtual disks.

2. Uninstall VMware Tools and then perform a clean shutdown of the guest operating system.

3. Copy the virtual hard disk(s) over to the qemu/kvm host. The virtual disk is typically the largest file within a VM’s directory and will usually be named something like ‘guestname-flat.vmdk’

4. On the qemu/kvm host, change to the directory containing the .vmdk file. Assuming you are using qcow2 disk images, run the following command to convert the .vmdk: kvm-img convert -O qcow2 guestname-flat.vmdk newguestname.qcow2

5. Create a new VM on the qemu/kvm host and choose the recently converted disk image as your existing drive/image. It is important that you create your new guest with the same or similar settings as it had before. I recommend cloning the MAC address over to the new guest for added simplicity with NIC detection, assignment, and third party software licensing.

6. Attempt to boot the system. Depending upon your guests virtual disk settings and other factors, the system may hang during boot. Edit your virtual machine and set the controller type to SCSI assuming that was the controller type back on ESXi.

At this point your system should be up and running on the new host. I did find notes and suggestions that qemu/kvm can run vmdk files/disk images, but there seemed to be a handful of caveats so I decided to convert the vmdk’s over to a native format.

–himuraken

So you like Debian, and why wouldn’t you, it is great after all. Unfortunately, many laptops come from the factory sporting Broadcom-based chipsets. So inevitably I complete a Debian install and Broadcom takes the wind out of my sales. I then trudge over to http://wiki.debian.org/wl#Squeeze and go through the paces. Why? I do it over and over. Well enough is enough, I mean this isn’t a tricky script to write. So for your enjoyment, I have put it all together into a small bash script to simplify things for future installs. First, be sure to add the non-free repo to your /etc/apt/sources.list file.
Then create and run a .sh file containting:

#!/bin/bash
aptitude update
aptitude install module-assistant wireless-tools
m-a a-i broadcom-sta
echo blacklist brcm80211 >> /etc/modprobe.d/broadcom-sta-common.conf
update-initramfs -u -k $(uname -r)
modprobe -r b44 b43 b43legacy ssb brcm80211
modprobe wl
iwconfig

Enjoy!

–himuraken

So several months ago, I like the rest of the world, was notified that end of life (EOL) for Ubuntu 9.10 Karmic Koala would happening. In the news blurb/mailing list, wherever I found it, I walked away thinking that security updates would cease to exist.

In preparation for the upgrade, I went ahead and cloned the 9.10 server and proceeded to upgrade the server to Ubuntu 10.04 Lucid Lynx. This went off without a hitch from what I could tell and I scheduled the upgrade of the production server with my last client running 9.10.

Without fail, life happens, clients have things come up, and the upgrade never happened. Fast forward to present day and time, and my client tried installing a package using apt-get and received a slew of errors. Looking into the issue a bit further and I found the repositories gone. Interestingly enough, when EOL occurs for an Ubuntu release, it really ends, and not just for the security patches.

So one is left wondering, “how can I sudo apt-get install update-manager-core & sudo do-release-upgrade when I can’t even do a simple sudo apt-get update?” Solution: EOL upgrade. There are several different ways to go about this, the best are detailed here. At the time of this writing, the link is a little unclear about how to get 9.10 to 10.04 so here is the quick and easy way:

1. Backup your current sources.list:
sudo mv /etc/apt/sources.list ~/sources.list

2. Create a new sources.list:
sudo vim /etc/apt/sources.list

3. Add/paste in archive release repositories substituting CODENAME for release jaunty, karmic, etc:

## EOL upgrade sources.list
# Required
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-updates main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-security main restricted universe multiverse

# Optional
#deb http://old-releases.ubuntu.com/ubuntu/ CODENAME-backports main restricted universe multiverse

4. Update repositories and install update manager
sudo apt-get update
sudo apt-get install update-manager-core

5. Initiate the upgrade
sudo do-release-upgrade

6. Enjoy!

–Himuraken

In my previous post on ZFS performance testing I ran through various tests on a particular test system that I had running at the time. That system has come and gone to a better place in the proverbial cloud. This go round, I have a similar server with a different ZFS configuration. Lets dive in to the system and tests.

Test rig:

• Dell PowerEdge 2950
• Xeon Quad Core 1.6GHz
• 8GB RAM
• PERC5 – Total of 5 logical drives with read ahead and write back enabled.
• 2x160GB SATAII 7200 RPM – Hardware RAID1
• 4x2TB SATAII 7200 RPM – Four (4) Hardware RAID0′s (Controller does not support JBOD mode)
• FreeNAS 0.7.2 Sabanda (revision 5543)-ZFS v13

GNU dd:
Tests performed from the CLI using good ole’ GNU dd. The following command was used to first write, and then read back:

dd if=/dev/zero of=foo bs=2M count=10000 ; dd if=foo of=/dev/null bs=2M

Results:
Results are listed as configuration, write, then read.

• Reference run: 2x160GB 7200 RPM SATAII RAID1
  • 85.6 MB/s
  • 92.5 MB/s

• ZFS stripe pool utilizing two (2) SATA disks
  • 221 MB/s
  • 206 MB/s

• ZFS stripe pool utilizing two (2) SATA disks with dataset compression set to “On”
  • 631 MB/s
  • 1074 MB/s

• ZFS mirror pool utilizing two (2) SATA disks
  • 116 MB/s
  • 145 MB/s

• ZFS mirror pool utilizing two (2) SATA disks with dataset compression set to “On”
  • 631 MB/s
  • 1069 MB/s

Notes, Thoughts & Mentionables:
There are a few things worth mentioning about this system:
On the hard disk side of things, the hardware RAID1 was made up of Western Digital Blue disks while the other four (4) disks are Western Digital 2TB Green drives. If you have done your home work, you already know that the WD EARS disks use 4K sectors and masks this as 512byte sectors so that OS’ don’t complain. If disks are not properly formatted and/or sector aligned with this in mind, performance takes a tremendous hit. The reason for such inexpensive disks for this build is simple: This server is configured as a backup destination and as such, size is more important than the reliability that a SAS solution would provide.

Compressions test results were, to say the least, quite interesting. It should be noted that the stripe and mirror pools performed quite similarly. Further testing of these results will be required, but it seems that the maximum score of 1074 MB/s was limited only by the CPU. During the read test all four cores of the quad core CPU were maxed. This becomes even more interesting when you compare the results of this two disk stripe pool with my previous findings on the six disk stripe pool running the same test. The earlier test rig scored much lower and it would appear to be the difference in CPUs that made such a strong difference.

–himuraken

***Update***
This started a simple post to share a few results from various levels of zfs/array testing. Be sure to check back from time to time as I add additional configuration results.

***Begin Original Text***
I have been playing around with ZFS on various operating systems lately and have been trying to compare performance. I figured that sharing some of my results would give others something to compare with. Plus, I am on borrowed time with this unit, it is big, loud, and taking up free space and spare time in the home office.

Test rig:

• Dell PowerEdge 2900
• Xeon Dual Core 3.0GHz (HT Enabled-OS showing 4 cores)
• 14GB RAM
• PERC5 – Total of 7 logical drives with read ahead and write back enabled.
• 2x146GB SAS 15K RPM – Hardware RAID1 for OS
• 6x1TB SATAII 7200 RPM – Six (6) SATAII 7200 RPM Disks for testing. (Controller does not support JBOD mode)
• FreeNAS 0.7.2 Sabanda (revision 5543)-ZFS v13

GNU dd:
Tests performed from the CLI using good ole’ GNU dd. The following command was used to first write, and then read back:

dd if=/dev/zero of=foo bs=2M count=10000 ; dd if=foo of=/dev/null bs=2M

Results: Each disk configured as a separate RAID0 array on controller.
Results are listed as configuration, write, then read.

• ZFS raidz1 pool utilizing six (6) SATA disks
  • 133 MB/s
  • 311 MB/s

• ZFS raidz1 pool utilizing six (6) SATA disks with dataset compression set to “On”
  • 414 MB/s
  • 359 MB/s

• ZFS raidz2 pool utilizing six (6) SATA disks
  • 180 MB/s
  • 286 MB/s

• ZFS raidz2 pool utilizing six (6) SATA disks with dataset compression set to “On”
  • 414 MB/s
  • 361 MB/s

• ZFS stripe pool utilizing six (6) SATA disks
  • 190 MB/s
  • 263 MB/s

• ZFS stripe pool utilizing six (6) SATA disks with dataset compression set to “On”
  • 429 MB/s
  • 381 MB/s

Results: Each disk configured as a member of a single RAID0 array.
Results are listed as configuration, write, then read.

• ZFS stripe pool utilizing six (6) SATA disks
  • 353.4 MB/s
  • 473.0 MB/s

• ZFS stripe pool utilizing six (6) SATA disks with dataset compression set to “On”
  • 420.8 MB/s
  • 340.9 MB/s

Results: Each disk configured as a member of a single RAID5 array.
Results are listed as configuration, write, then read.

• ZFS stripe pool utilizing six (6) SATA disks
  • 322.0 MB/s
  • 325.9 MB/s

• ZFS stripe pool utilizing six (6) SATA disks with dataset compression set to “On”
  • 438.8x MB/s
  • 371.8 MB/s

Results: Each disk configured as a member of a single RAID10 array.
Results are listed as configuration, write, then read.

• ZFS stripe pool utilizing six (6) SATA disks
  • 251.2 MB/s
  • 304.3 MB/s

• ZFS stripe pool utilizing six (6) SATA disks with dataset compression set to “On”
  • 430.7 MB/s
  • 360.9 MB/s

Notes, Thoughts & Mentionables:
It is worth noting that the results of the datasets with compression on can be a bit misleading. This is due to the source we are using with dd; /dev/zero. Feeding a string of zeroes into a compression algorithm is probably the best case scenario when it comes to compression. In real world conditions, data being read or written that is compressible would experience an increase in performance, while non-compressible data would likely suffer a penalty.

I am hoping to conduct the same tests on the exact hardware in the near future. I will be switching the six (6) SATA disks over to varying hardware RAID levels and comparing them again.

***Update***
In a follow-up post to this one, I concluded that compression read and write performance on this particular test rig was being limited by the CPU. I am hoping to swap out the current Intel Xeon 3.0GHz dual core for a quad core for additional comparison.

–himuraken

For those that have never used crontab, it can be a daunting task to schedule a simple script. As with all things, I highly recommend that you learn the meanings of CLI programs and their usage etcetera. With that said, I wanted to check my crontab job against a generator of sorts just to be sure I had it right. If you want a simple click and go on crontab generator, I recommend using the one provided by HTML Basix. Enjoy!

–Himuraken

Of course the first thing that I did tonight after receiving my 3ware 9650se was to install it in my ESXi 4.0 server and get it going. There are a few caveats that I expected and a few that I didn’t. Hopefully this post will help a few of you out there.

Here are the steps that I performed:
1. Physically installed card and drives.
2. Built the logical unit / volume on the card.
3. Booted ESXi and noticed the card / array not showing up.

I expected #3 from my pre-sales madness research, yeah I’m that bad about pre-sales. There is a driver for ESX 4 that enables the hypervisor to be installed onto the array. For the rest of us with ESXi 4.0 however, the array can only be used as a datastore. This isn’t an issue anyways since the critical files are on the array not the single disk or USB device that we install the ESXi OS onto.

This led me to 3ware’s support site to find the exact article regarding this. You can find the article titled “I need support for VMware ESX/ESXi 4.0 and ESX/ESXi 4.0 update 1 for 9650SE and 9690SA. Is a certified driver available?” here. Basically, the top half of the document applies to ESX while the lower portion is dedicated to ESXi.

First off, the fact that 3ware has this and other great articles is excellent, they seem like the “get it”. On the other hand I found something difficult to do, and not for technical reasons. They instruct you to find the file “offline-bundle.zip” on the included driver CD. I searched all over the provided driver CD and had no luck locating it; I couldn’t help but think that there ought to have been a download link as well***. After locating the file I proceeded down the list of the well written how-to. Unfortunately for me, after running the perl vihostupdate.pl -server x.x.x.x -username root -password "" -b c:\offline-bundle.zip -i command I received zero feedback from the CLI. I restarted the ESXi server per the documents recommendations and upon reboot, no RAID array. Hrmm, I didn’t get any errors or feedback. After shorting the name of the original file to offline-bundle.zip and re-running the command, I did get positive feedback in the form of this message: The update completed successfully, but the system needs to be rebooted for the changes to be effective. Delicious! Now were are working the way we ought to. Restart the ESXi server and enjoy.

*** – Clarification and special note: At the bottom of the page you will see a download link for a file named: vmware-esx-drivers-scsi-3w-9xxx_400.2.26.08.035vm40-1.0.4.00000.179560.iso.. This ISO image contains the offline-bundle folder. The file that you need for the upgrade/upload is named: AMCC_2.26.08.035vm40-offline_bundle-179560.zip. The process would not succeed until I renamed AMCC_2.26.08.035vm40-offline_bundle-179560.zip to offline-bundle.zip.

Happy virtualizing!

–Himuraken

This one is from a while ago. I originally used this back in May of 2009 and just realized that I hadn’t posted the link and info. Basically, shortly after migrating to Microsoft Exchange 2007 in-house, we purchased an SSL certificate for various reasons. Long story short, Outlook 2007 clients starting throwing certificate errors upon opening. So, our outside clients now trust the Exchange box, but the inside clients do not. I went round and round trying to resolve this on my own before searching the internet for a solution. Finding the solution was not easy at all, but I eventually located this link. After following the instructions, everything was up and running as one would expect.

I didn’t really think that this would have been such a PITA, but maybe thats what I get for naming a mail server Newman.

–Himuraken