KernelCrash

So I’ve had Scientific Linux 6.0 64 bit on my Thinkpad R60 for a while now, and for the most part it has been fine. But one thing that has really annoyed me is that everytime there is a kernel update, I notice that the laptop fan is stuck on (when the system is essentially idle). I always think “This can’t be right” and reboot back off my old kernel.

The thing is the only kernel I have now that has ‘good’ fan behaviour is 2.6.32-71.24.1 which is the one that went in when I first installed SL 6.0. Under that kernel , when the R60 is relatively idle, the fan will come on at 2500 to 2900rpm or so for say 60 secs, then be completely off for another 60 secs.  That’s kind of how I expect it to be when the system is idle (it would be nicer if the fan was off for longer periods … but sometimes off is better than never off)

A few yum updates later I also have 2.6.32-71.29.1 and the latest being 2.6.32-131.2.1. These two always run with the fan stuck on at 3300rpm or so (while the R60 is essentially idle). Very annoying. So it’s been interesting to read the various ‘linux power regression’ articles on Phoronix lately. The leading cause of the Recent Linux Kernel Power Problems article explains a key source of the problem related to some ASPM subsystem … and a boot option workaround; “pcie_aspm=force”

So I had a go at adding that option into the grub.conf for the 2.6.32-131.2.1 kernel and amazingly the fan behaviour is back to the “right” behaviour. The fan now comes on for about 60 secs at 2800rpm, then goes off for 60 secs and so on. Cool stuff. Many thanks to the Phoronix site for figuring this out.

Now, a key thing to remember with Redhat/SL kernels is that they contain many patches from more recent kernels ‘backported’ into an older kernel such as 2.6.32 … and hence that partly explains why this problem exists on the recent SL 6.0 kernels.

So, as per my previous post, I was searching for an upgrade path for my Thinkpad R60 virtualistation server running Debian Lenny. I’d tried upgrading to Squeeze a few times … and didn’t like it … always reverting back to Lenny.

So after trying a few different distros/virtualisation hypervisors (using a spare disk), I settled on installing Scientific Linux 6.0 64 bit. Key reasons are;

• It’s effectively Redhat 6.0, and Redhat ‘target’ stability rather than ‘latest and greatest’ which is what I’m after with a virtualisation box.
• By using KVM virtualisation on SL 6.0, it contains a version of libvirt that will let you disable the memory balloon (the one in Squeeze always has the memory balloon enabled). The memory balloon stuff is more annoying than useful on my little laptop server.
• It didn’t seem too annoying. Well that’s what I thought. Less annoying the Debian Squeeze I guess.

So I’d been trying out SL 6 on a spare disk … so now I thought I’d put it ‘properly’ on the two 500GB drives in the R60. I’d always run Lenny with a mirrored root and /boot so I wanted to do the same with SL 6.0. I’d already set split mirrors on the two 500GB drives. The original md devices I used on the first disk were md2 and md3. They currently had an old Debian Squeeze on them with a deliberately failed mirror. The other disk now had md20 and md30 which was the original Lenny install that I had ‘split off’ prior to my last Squeeze upgrade (md20 and md30 also being a degraded mirror). So I still wanted to keep md20 and md30 for now in case I wanted to go back to Lenny, and I would overwrite md2 and md3 with SL 6.0. I still wanted to put SL onto RAID1 mirror devices, because at some point I would kill md20 and md30 on the 2nd disk and properly mirror md2 and md3 again.

So I wanted to install SL 6.0 onto md2 and md3 which were both currently failed mirrors. Do you think it would let me do this? Nope. It seems the anaconda installer won’t let you do this … which is very very sad. To me it seems an obvious thing that you’d want to do. I found redhat bug 188314 which is basically a depressing read.

So, I dug out yet another spare disk (an old 30GB IDE) , removed the 2nd 500GB drive, repartitioned the 30GB drive to match the root and /boot partition sizes and then used mdadm to add in the extra partitions and eventually md2 and md3 were active RAID1 mirrors.

Then I tried to install again (lucky you can boot off a USB attached DVD drive on the R60). This time I got past the partitioning hiccup and had it installed. I used the SL 6.0 x86_64 LiveMiniCD … which doesn’t ask too many questions and gives you a reasonable (for 2011) size install. One thing on my R60 that I learnt from previous attempts, that at the inital GRUB screen press a key to get the boot menu, cursor down to ‘install’, press tab, edit the 2nd line, then add the word ‘nomodeset’ onto the end of the main kernel boot line. The ‘nomodeset’ turns off KMS and prevents the extremely annoying flickering I would previously get.

Once I had it installed, I then went about getting it sorted to work much the same as my Lenny setup. I’ll include some of my notes further down for reference, and just discuss some of the aspects of KVM setup right now.

So in order to get KVM set up right, I installed the main virtualisation stuff;

yum groupinstall Virtualization
yum groupinstall “Virtualization Client”
yum groupinstall “Virtualization Platform”

Then I had to edit /etc/libvirt/libvirtd.conf in order to get rid of the dependence on PolicyKit (or attempt to get rid of the dependence). Then I copied in the xml libvirt files from my Lenny install and started modifying them. I changed the kvm binary references to /usr/libexec/qemu-kvm and as I mentioned in my previous post, my Windows 7 32 bit VM always crashed on shutdown until I switched the cpu from i686 to x86_64 (as I had recently upgraded the CPU in my R60 from a 32 bit T2400 to a 64 bit T7200). The machine type in the xml files is also a puzzle. Lenny has a completely different set of them compared to Redhat/SL. Lenny had ‘pc’ as the main machine type, and on most of my attempted Squeeze upgrades, somehow the upgrade process seemed to think it was smart to change all my ‘pc’ machine references to ‘pc-0.12′. I remember noticing that this makes Windows think it has a new network card, new hard drive etc….

With Redhat/SL the QEMU/KVM machine types are pc, rhel6.0.0, rhel5.5.0, rhel5.4.0. It says ‘pc’ is the same as ‘rhel6.0.0′, so if I’d left my libvirt xml files all with ‘pc’ they’d be automatically upgraded … just like Squeeze did.  I think I currently have rhel5.4.0 as the choice for my Windows 7 VM for lack of a better choice.

Anyway, in the process of getting Windows 7 going again, I got the dreaded ‘Windows Actiation error’. Sure it now thought that not only had the network card changed, the hard drive and now it thinks it has a new CPU. I thought one of the ‘features’ of virtualisation is to have the ability to have a ‘constant’ hardware abstraction.

So I try to activate Windows 7 online. That fails, so I have to ‘ring MS$’ which is definitely the low point of the week. I type in a very long string of numbers on the phone keypad, only to be told that I needed to be put through to an operator. I read the same long series of numbers to a human, who then reads a similarly long series of numbers back to me … which finally does the trick.

I’m thinking at this point … that maybe I should listen to all the people that comment on my ESXi posts and ditch KVM and just switch to ESXi.;-)

But I am a glutton for punishment.

Anyway, it is all kind of working OK now. I added in the line to disable the memory balloon for my KVM guests, and also found there was some ksm daemons that also needed to be disabled, as they sporadically ate lots of CPU as well.

My notes;

# Get rid of the leftovers from doing the LiveCD install
chkconfig --del livesys
chkconfig --del livesys-late
# Turn SSH on
chkconfig --level 345 sshd on
service sshd start
# Turn NetworkManager off.
chkconfig NetworkManager off
service NetworkManager off
# Updated eth0 and br0 network configs

edit /etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-br0
chkconfig --level 345 network on
service network start

# The LiveCD never seems to set the hostname
set HOSTNAME in /etc/sysconfig/network
Add our own hostname and IP to /etc/hosts
Comment out hiddenmenu in /boot/grub/menu.lst. Also add in my Lenny boot menu item
# Fix up some sudo rules
visudo
sudo yum upgrade

# Add in all the virtualistion stuff
yum groupinstall Virtualization
yum groupinstall "Virtualization Client"
yum groupinstall "Virtualization Platform"

# Turn off ksm
chkconfig ksm off
chkconfig ksmtuned off
service ksmtuned stop
service ksm stop
#LiveCD leaves firewall on, so need to do some updates to /etc/sysconfig/iptables and restart the iptables service

#I installed the development tools and some other stuff

yum groupinstall "Development tools"
yum install libX11-devel
yum install sg3_utils
yum install expect
yum install libXfont-devel

#downloaded EPEL release 6.5 rpm to get EPEL set up

#Compiled X11rdp and xrdp from svn and cvs respectively. Needed automake1.9 for  X11rdp. Downloaded a srpm for it and compiled by hand.
xrdp 0.6 has changed a bit. Still has the bug where it eats heaps of CPU though, so added the changes mentioned on this page;

http://sourceforge.net/projects/xrdp/forums/forum/389417/topic/3706601

#Quite a few deps to handle to do the xrdp compile. Can't remember all of them
yum install pam-devel
yum install libXfixes-devel

So, my main virtualisation machine here at home is a Thinkpad R60 running Debian Lenny, with KVM as the virtualisation engine. This has been my main ‘server’ host for over a year and a half now. I have quite a few VMs on it, but most are for testing out stuff, so there is only one or two VMs that are active constantly. One is a Debian Lenny guest running Asterisk for the most part. The other is a Windows 7 VM. I’ve been reasonably happy with this setup. Key positive points to me are;

• I can use the host as a regular linux box with a graphical desktop
• Performance is OK
• The Debian Lenny guest is idle most of the time, and is lucky to consume 2 or 3% of the host cpu.
• It all works with a 32 bit VT capable CPU such as the T2400 core duo in the R60

Negatives of this setup are;

• The T2400 is only 32 bit so no 64 bit guests.
• Physical host CPU use when the Windows 7 guest runs is high; 14% or so. I eventually tracked this down to the USB tablet driver. Once removed, host CPU use for the related kvm process is about 2 to 3% for an idle Windows 7.
• You cannot do easy snapshots and reverts like you can say with Vmware’s products.

So Debian Squeeze is the new stable ‘Debian’, so I have attempted to upgrade my system. I’ve actually done this now several times, first using some of the ‘pre stable’ releases of Squeeze, as well as the current stable Squeeze. Each time I have reverted back to Lenny. Key negatives of Squeeze are;

• Host CPU use seems higher for all guests for me.
• The extra CPU use by the USB tablet driver in a Windows guest still seems to be there. Makes a big difference if you remove it.
• The Windows 7 guest now seems to BSOD (ie. crash) when you shut it down (I’m just using a virtual IDE disk and a virtual RTL3129)
• The Laptop fan seems to stay on more often now.

So this has had me looking at alternatives. I am a big fan of Vmware ESXi, but because of the 32 bit CPU in the R60 I am limited to ESXi 3.5 on it. With that in mind, I did my research and discovered that you can put a T7200 core 2 duo CPU into my R60 which is a 64 bit VT capable CPU. So I ordered one off eaby and have had it running in it for a few days now.

So here’s a quick breakdown of what I’ve tried lately. Some of these I’ve only set up on my desktop core 2 duo system, but I intend to try them all on the R60

• Squeeze 64 bit. Much the same as the problems I’ve had with Squeeze mentioned above.
• ESXi 3.5u5. This works pretty well. On the R60, all the hardware works, except when you do a shutdown on the console, you get a purple screen of death. Reboots are OK though.
• ESXi 4.0 (I will try 4.1 shortly, but I had the 4.0 CD handy). That works pretty good too. The main negatives for ESXi on this box are; 1) no graphical console anymore, 2) one disk has to ‘generally’ be set aside as ESXi vm storage, 3) I have to use Windows to administer it, 4) I can’t read any of the Thinkpad thermal or fan information.
• Scientific Linux 6.0 64 bit. I tried the ‘LiveMiniCD’ which is pretty good. It installs an IceWM desktop. I’ve been running my Asterisk VM and my Windows 7 VM on it under kvm. In this case CPU use for both is low, but the Asterisk VM CPU use is still quite a bit higher than my Lenny setup. Also the Windows 7 VM crashes on shutdown just like under Squeeze. Scientific Linux 6 and Squeeze both use a very similar version of qemu-kvm. I might persists a bit of Scientific Linux. It doesn’t seem too bad (for a Redhat clone). One problem I had with SL 6.0 was a strange flicker on the X desktop every 10 secs. Changing the GRUB kernel args to include ‘nomodeset’ (ie. turn off KMS), cured this.
• Xenserver 5.6 FP1 . I haven’t tried it on the R60, just my desktop core 2 duo box. This I thought worked pretty well … and Citrix have done a lot to make it ‘feel’ just like ESXi for the most part, and the Windows admin tool seems to let you do most stuff that Vmware’s admin client let’s you do. I did have problems installing the guest tools into a Windows 7 guest though. This was a week or too back and I can’t quite remember what it was. But I was impressed by it in general, and I’d like to spend more time looking at it (or XCP below)
• XCP 1.0. This to me looked to be almost exactly the same as Xenserver 5.6 FP1. It had exactly the same problems re the guest tools in Windows 7.

So still undecided at the moment. I need to do a bit more research on the ESXi front. The R60 can only handle 3GB of RAM, and I think the newer ESXi’s have quite a large default memory footprint (but I think there is a way to reduce it a bit). If I could sort out the problem with the Windows 7 BSOD on shutdown then I might consider Scientific Linux. Lot’s of options.

UPDATE (2011.5.2). The Windows 7 BSOD I was getting when performing a shutdown on Scientific Linux or Squeeze was a IRQL_NOT_LESS_OR_EQUAL. I think I’ve found what causes this. All my VMs are created with i686 as the CPU type in the libvirt xml files (since my old T2400 chip was 32 bit only). Of course now I have a 64 bit capable chip (T7200) and the new versions of qemu-kvm don’t seem to like me specifying a i686 for my Windows 7 32 bit instance. If I change the xml file to use x86_64 instead of i686, I can now (finally) gracefully shut the Windows 7 32 bit VM down.;

<type arch=’x86_64′ …

Another useful thing to keep my CPU utilisation low is to disable the memory balloon (which I discovered is enabled by default in more recent libvirt’s). You just need to add this to the <devices> section of your libvirt xml file;

<memballoon model=’none’/>

Yes, I know Snow Leopard has been out for ages, and I did actually purchase the ‘real’ install DVD for it some time ago, but well ‘Leopard’ seemed to be working fine on my Macbook, and well I’m very cynical of these claims that ‘upgrading is easy and trouble free’. However, I bought a Hitachi 5K500.B 500GB drive recently as either an upgrade for the 160GB drive in the macbook or as a replacement for the original 80GB drive in my R60.

Given that I had a new (larger) drive. I thought I’d play with how well Time Machine and the Migration Assistant work. So as a test I setup a large mount point on a linux box and shared it out using Samba to the Macbook, then did the defaults command below to enable a samba share as a Time Machine destination:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

I also did the mucking around with hdiutil to create the necessary sparsebundle (if you google for ‘TMShowUnsupportedNetworkVolumes hdiutil’ you’ll find a heap of references). It takes a long time to do the first backup. I then set it to ‘manual’ mode so it’s not doing continual backups. Then I did one last Time Machine backup and shut the Macbook down (running Leopard).

Now I swapped in the 500GB drive. Put in the snow leopard DVD, held down ‘c’ and let it boot up off the DVD. Then formatted the drive, and installed Snow Leopard (I picked a different account name than the one I used on my Leopard install, as Migration Assistant will complain later on if you try to restore into the account you are currently logged in as). Once I had SL installed, I then did a ‘connect to server’ in finder to mount my Samba share, then you need to click on the sparsebundle that you see, and your time machine backup should now be mounted. Now you can run ‘Migration Assistant’. Tell it you want to restore from a ‘Time Machine’ backup, then you should see your time machine backup to choose, then you get some screen with a few tick boxes and you have to wait while it says ‘Calculating’. I just left everything ticked and let it chug away restoring (it takes some time). Eventually, I just restarted, logged in as the username I used to use and voila it all looked suspiciously like my old Leopard setup.

The first thing I wanted to make sure was working was Mail. I started it up and it gives you some messages saying it needs to re-import everything and away it goes. Now, I use Mail.app as my main mail client, and I have an IMAP server that it connects to. I have some anal security in place for the IMAP server, so when Mail was restarted after the re-import I didn’t let it connect properly to the IMAP server. I then started going through my Inbox to see whether it looked OK. Everything seemed to be there, but I had a weird problem when I clicked on attachments. They would never load. There’d be some delay and then a popup highlighting there was a permission problem. This led to much trawling through Apple discussion threads and trying various things, but I could not get attachments to display. I just received that error instead

So I switched back to the original 160GB drive for a while. Then I tried again, clean format of the 500GB drive, installed SL, then this time I upgraded to 10.6.2, then did the Migration Assistant restore. No luck. Still had the same permission problem.

So by this time I was getting frustrated with Snow Leopard, so I thought I’d try to just put Leopard on the 500GB drive. Rather than use something sane like Carbon Copy Cloner or similar I thought I’d see how good a full restore from ‘Time Machine’ would go onto the 500GB drive. So with the old 160GB drive containing leopard, I did another Time Machine backup to the Samba share, shut down, swapped in the 500GB drive and booted off the Leopard install DVD that came with the macbook. Booted off the DVD, formatted the 500GB drive, then chose the option to restore from backup. This is where I learnt that the Apple install DVDs cannot mount a Samba share (or any kind of smb based share). They can mount a afp share though. So I thought, I’ll install netatalk on my Debian Lenny server that was running Samba, and just get it to share out the same mount that Samba was sharing.

This is when I discovered that the Debian Lenny version of netatalk cannot deal with encrypted passwords. Guess what OSX will only connect to? Yep, it only connects to a netatalk server that can do encrypted passwords (there is actually a defaults ‘switch’ that will allow you to connect to a netatalk user ; defaults write com.apple.AppleShareClient afp_cleartext_allow -bool true, but alas I could not get that to work from the shell environment offered on the install DVD), so then I ended up recompiling netatalk on Debian Lenny to handle encrypted passwords, which goes somehting like this;

$ sudo apt-get install libdb-dev
$ sudo apt-get install cdbs
$ sudo apt-get install libcrack2-dev
$ sudo apt-get install dh-buildinfo

$ cd debtmp
$ mkdir debtmp
$ sudo apt-get source netatalk
$ sudo apt-get install libcups2-dev
$ sudo apt-get build-dep netatalk
$ cd netatalk-2.0.3
$ sudo DEB_BUILD_OPTIONS=openssl fakeroot debian/rules binary
$ sudo dpkg -i ../netatalk_2.0.3-11+lenny1_amd64.deb

sudo afppasswd -ac
sudo afppasswd myuser

You might need to tweak some of the other files under /etc/netatalk too.

OK, so now using the notes on mounting here; I could mount my Time Machine backup from the Leopard install DVD;

mkdir /Volumes/tmachine
mount -t afp afp://username:password@server.hostname/tmachine /Volumes/tmachine

So then I kicked off my restore. Maybe 5 or 10 mins later the restore had encountered an error. I honestly can’t remember what it was. The discussion threads I found when googling for the error hinted that it has something to do with using the wrong install DVD (ie. one that came with a different Mac). I was definitely using the DVD that came with my Macbook, but I thought since I had the Snow Leopard DVD as well, then surely a full restore of a Leopard backup using the Snow Leopard boot DVD would end up with a Leopard system. So I did that, mounted the afp share OK, then started the restore … which continued just fine. Odd.

So that chugged away, and eventually I had a Leopard system on a bigger drive. I opened Mail.app, and again it said it needed to reimport Mail (just like for Snow Leopard). It chugged away for a bit, and then I had my Inbox open and it looked intact. I then found some of the messages with attachments I had had trouble with before and tried to open them. Instead of getting a permission error, I received no message at all … and in fact nothing happened (ie. my attachment still did not display). Hmmmm. Odd. It’s about now that I somehow realised that by blocking access to my IMAP server, I was preventing it from grabbing the attachment. This is probably obvious, but I imagine that Mail.app downloads attachment when you first click on them, and then caches them in the ~/Library/Mail\ Downloads directory, so that normally you’d be viewing the cached copy. However, I know that Time Machine ignores most cache directories in order to save space … so after my ‘fresh’ install plus restore, the cache directory was empty. I guess I was able to confirm my theory quickly by simply allowing my connection to my IMAP server to work (I noticed after I did this that there is a lot of activity in the ‘Activity’ window regarding caching etc.)

So after all that hoopla about Mail attachments, I thought I’ll reinstall Snow Leopard again. So same deal as before; boot off the SL DVD, format the 500GB drive, let it install SL, get it booted, perhaps upgrade to 10.6.3 or whatever, then use the Migration Assistant to restore from my Time Machine backup. This time when I ran Mail.app I let it connect to my IMAP server, and attachments worked as expected. All good

Apart from my funny Mail.app issue (which was sort of my fault), I was quite impressed with the whole Time Machine backup/restore. I am often asked by family or friends about ‘backing up their Windows PC’, and well there are a bunch of ways to back things up … but I always think a backup is only as good as your ability to restore it. My thoughts were that I could probably talk someone through doing a full restore of a Mac using a Time Machine backup … over the phone (maybe not from my convoluted AFP/Samba shared Time Machine server, but from a local USB drive). I am sure there are Windows products that claim to do something similar, but an enticing thing is that Time Machine is pretty much supported ‘out of the box’.

My upgrade was not all good. I have several vmware fusion guests on my Macbook, and when these were restored on the 500GB drive, all of them seemed to have very scary filesystem errors (the ones you get when there’s been some serious corruption). I thought maybe this was to do with Time Machine, but I ended up using my crazy rsync backups to restore from as well … and I still had the same filesystem errors. In the end, I booted back up off the 160GB drive in the macbook, and went through every Vmware guest and told them to power down (pretty much all of them were just ‘suspended’), then I did my rsync backup again, and restored them with the 500GB drive inside. And now they all work and have no filesystem errors. Obviously there is some strange inconsistency that occurs with the guest ‘state’ information during the upgrade to Snow Leopard.

One thing that would be nice to have in linux LVM is the ability to take a snapshot of a logical volume, make some changes then ‘roll back’ to the state preserved in the snapshot. If you look at the current set of lv commands on most linux distros there is no such option to ‘revert to snapshot’. Now, that I’m using KVM more, the ability to ‘rollback’ a virtual machine installed on a logical volume would be particularly handy.

Well it turns out you can ‘revert to snapshot’, but you just have to use a non-standard tool and some dmsetup commands. The tool is called dm-merge. Go to the dm-merge git repository or download the latest dm-merge snapshot . dm-merge just looks at the ‘changed blocks’ and pushes the old copies of them back into the original logical volume. You’ll need to compile the dm-merge program (just do a ‘make’). dm-merge comes with some doco. It explains two possible methods; reverting with an active logical volume OR reverting with an inactive one. In the former case you’re attempting to revert to a snapshot when the filesystem you’re trying to change is already mounted. I imagine that to be a very dangerous activity (but I guess it works well enough if there is little filesystem activity). I chose the latter (safer) method.

Here are my notes on the procedure. If this were a document for work, I would a) tell people they were crazy to even attempt this and b) if you’re still crazy make sure you make a backup. At least try this procedure out on a test machine first. The possibility of screwing your disk by virtue of a small typing mistake is quite high.

In the example below, I had a logical volume /dev/vgz/lvubuntu910full32bit that I was using as my hda drive in a KVM guest machine, and I had created a snapshot ‘snap1′ beforehand;

   lvcreate --size 1G --snapshot --name snap1 /dev/vgz/lvubuntu910full32bit

One important thing I realised recently is that the amount of space you allocate to a snapshot (eg. 1GB in the example above) can actually be resized without killing the snapshot (eg. lvresize -L 20G /dev/vgz/snap1)

Most of the following is a rehash of what is in the README that comes with dm-merge. I’m only referring to the method involving an inactive logical volume

# Firstly, shut down access to your logical volume (ie. /dev/vgz/vgzlvubuntu910full32bit).
   Since this is a KVM guest, all I did was shut down the guest. If you're doing this on a
   mounted logical volume then you should umount it.

# Duplicate the tables of the logical volumes and snapshot storage (note the use of the
   special -real and -cow suffixes)

dmsetup table vgz-lvubuntu910full32bit-real |dmsetup create tmplv
dmsetup table vgz-snap1-cow |dmsetup create tmpcow

# Deactivate the main lv

lvchange -a n /dev/vgz/lvubuntu910full32bit

# flush buffers

blockdev --flushbufs /dev/mapper/{tmplv,tmpcow}

# Do a dm-merge test run. You see lots of lines that start with 'dd' and a summary at the bottom.

dm-merge -i /dev/mapper/tmpcow -o /dev/mapper/tmplv -vd
...
dd of="/dev/mapper/tmplv" seek=2135444 if='/dev/mapper/tmpcow' iflag=direct skip=8275 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=2135445 if='/dev/mapper/tmpcow' iflag=direct skip=8276 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=2135446 if='/dev/mapper/tmpcow' iflag=direct skip=8277 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=5056466 if='/dev/mapper/tmpcow' iflag=direct skip=8278 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=5056467 if='/dev/mapper/tmpcow' iflag=direct skip=8279 count=1 bs=8b
Found 8246 exceptions of chunksize 4096, total size 33775616 bytes (32984 KiB, 32.211 MiB, 0.031 GiB).

# Do it for real. You get a couple of lines of output plus the same summary line.

 dm-merge -i /dev/mapper/tmpcow -o /dev/mapper/tmplv -f

Artificial sleep (1 second)
Found a proper MAGIC header: 0x70416e53
valid = 1
version = 1
chunk_size = 8 (4096 bytes)
Found 8246 exceptions of chunksize 4096, total size 33775616 bytes (32984 KiB, 32.211 MiB, 0.031 GiB).

# Flush buffers again

blockdev --flushbufs /dev/mapper/{tmplv,tmpcow}

# Remove the tmp lv's. These are just references to data, so it won't delete your real data.

dmsetup remove tmplv
dmsetup remove tmpcow

# Activate it

lvchange -a y /dev/vgz/lvubuntu910full32bit

Now try to start your KVM guest again … or remount it to check it out and confirm that it’s reverted to the original state. If you snapshotted a live KVM guest or a mounted filesystem, then you may find that your KVM guest will do a filesystem check on boot (fair enough), or you may need to run fsck on your filesystem before you can mount it.

Don’t forget to remove the snapshot you initially made (eg. lvremove /dev/vgz/snap1)

One thing to note is the ‘revert to snapshot’ capability has been ‘coming’ for a while in mainstream LVM. There’s a recent mention on LWN about it

For some time now, my old Thinkpad T42 has been running as a lightweight server at home. Some people think I’m a bit mad using a laptop as a server, but my vague reasons are a) they’re generally quiet, b) they don’t use a heap of power, c) the battery is in some ways a built-in UPS and d) whenever I’ve tried to take a regular desktop system and make it quiet I end up spending lots of $$$ on various quiet fans and power supplies only to be moderately satisfied with the results. To me, an old 2nd hand laptop is generally better value. Of course, you can’t fit many disks into laptops, but with the T42 I discovered that even though the primary drive has to be a 2.5″ IDE drive, you can fit a large 2.5″ sata drive into them by buying a sata HDD caddy (look on ebay).

Anyway, I have Debian Lenny on the T42 with several OpenVZ containers for things like Asterisk and a mercurial server. In addition it’s a streaming server, as I have two internal drives plus a large external 3.5″ USB drive.

The T42 is getting old, and I’d like to be able to run KVM on it … because basically I like KVM, but KVM needs a VT capable CPU. So I kept my eye out for a cheap 2nd hand VT capable laptop. I ended up buying a 2nd hand Thinkpad R60. This is probably one of the ugliest Thinkpads ever made which might explain the good price I got it for. I certainly wouldn’t buy one to lug around, but sitting on a desk at home quietly running a number of VMs is all I really wanted. It has a T2400 core duo (not core 2) cpu. Technically it is ‘VT capable’, but its a bit of an odd one (which probably applies to the T2500, T2600 etc) in that its not a 64 bit CPU like most other VT cpus. Other than being ugly and having an odd CPU, this R60 does have a lot going for it; 3GB of RAM, DVD burner, firewire, wifi, bluetooth, cardbus and expresscard slots (so I could add an esata expresscard device to add some highspeed desktop drives later on). And the HDD caddies I had for my T42 also fit in the R60.

I originally tried Ubuntu 9.10 on the R60, but the version of qemu they use in combination with KVM actually does not work properly on this old T2400 CPU. I could seemingly install a guest OS, but after reboot I had errors galore. Something to do with the ‘No execute’ bit I think. Anyway, this was yet another case of ubuntu quickly irritating me, so I ended up installing something else. Back to Debian Lenny. And of course VM’s work fine under Lenny and it’s older KVM and older qemu tools.

So far I’ve just run debian lenny 32 bit and Windows 2008 R2 32 bit in some KVM VMs. Both seem to run quite well. The laptop has 3GB of ram, so I can run quite a few VMs at the same time OK. I’ve migrated my asterisk installation from the openVZ container it was in on my T42 to a KVM VM on the R60. It works quite well.

One thing that does bug me about using KVM is the lack of easy to use snapshots. ESXi spoils you in the simplicity of it’s snapshots on running or shutdown VMs. You can use them for backups or easily roll back to them. However, KVM has a haphazard collection of tools that don’t quite do the same thing. virt-manager offers no option for snapshots. If you use virsh instead it does have a ‘save’ option to snapshot a machine … but it seems to always shut the machine down when it runs. If you don’t use libvirtd to manage your VMs and instead use kvm directly, then you can interact directly with the qemu monitor of a VM and it has a savevm option which does something akin to a snapshot, but everytime I’ve tried it on a live VM, it pauses the VM for several minutes which is somewhat annoying. The closest thing to ESXi snapshots that I can think of is to use logical volumes for the VM’s virtual disk, then use LVM snapshots to create snapshots of them. These aren’t full ‘state’ snapshots that include a ram snapshot etc, but are probably good enough for my purposes. The tricky part is rolling back to a snapshot which is not really supported with logical volumes out of the box, but you can do it with something like the dm-merge tool.

I also tried Fedora 12 on the R60. The motivation was that the version of KVM on Debian Lenny is old, and Fedora 12 has quite a number of new features of KVM including better qcow2 performance and something that looks for identical memory pages in VMs to reduce memory usage. This latter feature appears as some ksmd daemon that seems to hog 50% of one CPU. Very annoying, so I killed it. I was hoping that the qcow2 enhancements might solve my ‘snapshots’ issue, but so far it doesn’t look like it. I need to investigate more. In the meantime, I’ll go back to Debian Lenny on the R60.

Of course, I did try running ESXi on the R60. ESXi 3.5u4 seemingly worked perfectly on it. I didn’t't need any fancy DIY network drivers. It just picked up the internal broadcom gigabit and was able to see the two internal drives. The only problem is that I get a Purple screen of death when I try do a shutdown (some error to do with ACPI, and setting acpi=off in the kernel options does not help). However, if you tell ESXi to reboot it reboots as expected (UPDATE: I have just tried ESXi 3.5u5 … and is has the same purple screen when you attempt a shutdown). I didn’t bother trying ESXi 4, as I am pretty sure that needs a 64 bit CPU.

My main linux box here is about 3 years old now. The case is older, but I bought the cpu, motherboard, graphics card and some of the ram when the first Core 2 Duo’s were announced. It was expensive at the time, but it was well worth it. I bought a 1.86GHz E6300 C2D (not to be confused with the more recent 2.8GHz Pentium E6300 that Intel make).

The motherboard was a Asus P5LD2 which was one of the few ‘cheaper’ socket 775 boards available at the time that were compatible with the (then) new Core 2 Duos. It’s an interesting board as it has three IDE channels (ie. up to 6 IDE drives) and four SATA II interfaces. It uses a 945P chipset which doesn’t have a great reputation for overclocking. However, I could get the E6300 to run at 2.33GHz reliably which was ok.

More recently I’ve added more ram to the board ( four 1GB DIMMS in total) and discovered the problem of some of these older socket 775 chipsets. They can really only have 3GB maximum RAM. I thought this was just a 32bit OS problem, but 64 bit OS’s have the same problem with this board. Basically the chipset can only address 4GB of memory, and quite large chunks of memory are mapped for various IO purposes, which is made worse by certain types of cards (I think I get 3.5GB if I pull out my TV tuner card, 3 or 3.2GB with it plugged in). I know some older motherboards have an option for remapping a memory hole … but even on the latest BIOS, I don’t have such an option.

Also, the board is limited to mostly older core 2 duo chips, and quad cores are definitely a no go.

So, like any tech nutter, I’ve been researching the best ‘bang for buck’ upgrade. All the Intel socket 1156 i5/i7 chips came out a month or two ago and they do indeed look quite interesting. The i5 750 is quad core, but all the i7 models have a newer form of hyperthreading so it is a bit like getting ’8 cores’. To upgrade my system to i5/i7 there is quite a bit of expense (close to NZ$1000 just for an i7 860, motherboard and 4GB of ram), so I wondered exactly how much faster they really are. A good tool I found is the anandtech bench (beta).

With the Anandtech bench you can compare benchmarks between any two CPUs that they happen to have reviewed. A good test I thought would be to compare the i5 750 against an older quad core like the Q8400. The i5 is definitely faster in most tests, but in some of the tests that I’m interested in (like the x264 tests) it’s not ‘amazingly’ faster.

So feeling quite ‘cheap’ I thought I’d just upgrade my motherboard to a newer socket 775 board. That way I could keep my DDR2 ram, keep my core2duo and immediately get a full 4GB of ram, plus the ability to get a quad core core2duo later on. Sounded reasonable at the time. So I did some research and ended up coming home with a Gigabyte EP45-UD3LR board.

Supposedly the EP45-UD3LR can handle 16GB of ram … but until someone makes cheap 4GB DDR2 DIMMs, I’ll be happy with 4GB. It’s a full ATX board and has no onboard gfx. Just sound and the usual plethora of usb and sata ports. Now, I usually run linux on this system, but a secondary reason for getting this board is that these ‘EP45′ boards are very popular in the Hackintosh community for turning regular PC’s into ‘hacked’ OSX systems. I don’t really need another Mac, but all this hackintosh stuff is quite an interesting diversion in my pursuit of time wasting activities.

So for the past few weekends I’ve been playing with installing Snow Leopard on this thing (I actually bought the retail DVD for it). It’s still a far from simple process, that requires reading copious Howto’s and forum posts on InsanelyMac and other related sites. There are a number of complications; a) All the Gigabyte EP45 boards are slightly different, so while lifehacker.com might publish some nice howto’s for someone using a EP45-UD3P motherboard. My EP45-UD3LR is not ‘quite’ the same b) There are different ‘approaches’ you can take that have bizarre names that unless you’ve been trawling forums for quite some time you won’t understand; you can use hacked kexts, you can modify the DSDT, you can ‘inject’ EFI strings and seemingly you can use a combination of these .. though I’m not entirely sure. I think what’s happened is that the hackintosh community is obviously trying to increase the probability that when you do an Apple ‘Software Update’ that includes a kernel update (eg 10.6 to 10.6.1) that your machine doesn’t die painfully. I got the idea that modding the DSDT instead of injecting EFI strings somehow makes things ‘better’ and a big goal is to reduce the necessity for hacked kexts.

So does it work? Yes it does. A lot better than I thought it would even though I have had intermittent crashes (the screen goes dark and you get a message telling you to hold in the power button) which I think I’ve finally resolved (my Nvidia card seems to hate the 64 bit kernel drivers, but works OK booting in 32 bit mode). I won’t go into all the detail of what I did to build the thing, but here are my key points;

• I used the lifehacker.com ‘How to build a Hackintosh with Snow Leopard Start to Finish’ article (the original article, not the followup)
• I also referred to this EP45-UD3R guide on insanelymac. Especially the bits about setting the UUID in /Extra/smbios.plist and the PlatformUUID.kext. I found I couldn’t install Chameleon onto the hard drive until I got the UUID fixed
• I have an old GeForce 7300LE card with 128MB of RAM. This card is not listed in EFIStudio, but originally I just selected GeForce 7300 GT 256MB which seemed to work OK. I also noticed the Front Row would not work at all unless I had something like the following in /Extra/com.apple.Boot.plist:

      <key>Graphics Mode</key>
      <string>1280x1024x32</string>

• I used the ‘SL Pack’ that is referred to on various insanelymac howtos, especially the Extras folder it tells you to start with. ie. I didnt use the one that the LifeHacker article refers to. Though I did originally use the lifehacker one. I have no idea if there is any difference.
• To get my ALC888 sound to work required quite a few steps. There’s a really useful comment by Dith close to the bottom of this forum page. I followed the steps that start with ‘Redid my whole Snow Leopard installation’. And then I grabbed the LegacyHDA.kext that is at this path in the SL Pack and copied it into /Extra/Extensions replacing the one I already had.;

  SL Pack/DSDT Stuff/How to Patch DSDT /series of LegacyHDA 888 (ALC888)/3out2in HDA headphone/LegacyHDA.kext

• I manually enable QE/CI (well I think I did) by sudo’ing to root and running;

  defaults write /Library/Preferences/com.apple.windowserver GLCompositor -dict tileHeight -int 256 tileWidth -int 256

• I tended to run ‘Kext Utility’ between a lot of the above steps. Never sure if it makes much difference.
• I later found that OSX86Tools can generate the EFI string for my GeForce 7300LE 128MB card. So I used that in my /Extra/com.apple.Boot.plist hoping that it would fix my intermittent crash problem. It didn’t fix the crashes, but it did make me feel warm and fuzzy.
• I found that I could consistently crash the machine by running ‘Chess’ and just moving the mouse. Obviously Apple knows that I’m not very good at Chess, and was saving me some misery … These crashes were with a 64 bit kernel. Then I tried booting with a 32 bit kernel and Chess works fine, and so far no crashes (though I still can’t play Chess very well). I’m using Chameleon RC3 and had to put ‘arch=i386′ in the /Extra/com.apple.Boot.plist file like so;

          <key>Kernel Flags</key>
  	<string>arch=i386</string>

I’ve since updated from 10.6 to 10.6.1 with no dramas. The upgrade to 10.6.2 had some minor hitches in that you need to remove the older SleepEnabler.kext. See here for more info.

And of course, the new motherboard works fine running linux too!

I’d read about quite a few new features in the linux 2.6.31 kernel,so I thought I’d download the official source for 2.6.31 from kernel.org and build a custom kernel on my Debian Lenny 64 bit core2duo system. Thats the usual make-kpkg melarchy which takes an eternity.

It took me a while to get 2.6.31 to boot properly. I have an older nvidia card in this system, and the default setup had the kernel loading the nvidiafb module … which does not seem to work with my system (I just get a black or gray screen). Fortunately I could still ssh in from another box. It took me a while to work out how to properly disable nvidiafb, which ended up being somehting like;

cd /etc
mv modules.conf modules.conf.orig
cd modprobe.d
# Edit the blacklist file and add 'blacklist nvidiafb' to the end.
vi blacklist
# now update the initrd
update-initramfs -u

Also, in order to get X windows working I ended up downloading the driver from nvidia.com ( I used NVIDIA-Linux-x86_64-185.18.36-pkg2.run).

Once I had it booted (properly) off 2.6.31, I thought I’d have a look at btrfs. It’s a new filesystem for linux which is meant to be a lot like ZFS, but it’s not quite there yet in the stability department. However, I thought I read somewhere that the version included in 2.6.31 was ‘pretty good’, so it was worth a look. What I wanted to try was to have a btrfs root partition.

The first thing was to recompile my kernel again. Btrfs seemed to be turned off by default if you’re doing the make-kpkg stuff. So my setup process was something like;

cd /usr/src
tar xvjf /tmp/linux-2.6.31.tar.bz2
mv linux linux.orig
ln -s linux-2.6.31 linux
cd linux
make-kpkg clean
make menuconfig  <--- make sure you go to filesystems and enable btrfs as a module)
fakeroot make-kpkg --initrd --append-to-version=-custom2 kernel_image kernel_headers
# Wait 10 years
cd ..
dpkg -i linux-headers-2.6.31-custom2_2.6.31-custom2-10.00.Custom_amd64.deb
dpkg -i linux-image-2.6.31-custom2_2.6.31-custom2-10.00.Custom_amd64.deb

Now my plan was to make a copy of my root fs into a btrfs partition and boot off that. I have /boot as an ext2 partition, and my normal lenny root fs is on a logical volume as an ext3 fs. I thought I'd take a snapshot of the ext3 rootfs and copy it into a btrfs logical volume and boot off that. But the first thing to do is to get the btrfs user space tools;

git clone git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs-unstable.git
cd btrfs-progs-unstable
make
make install

Then setup the logical volumes and copy the data (NB: You might have to do a 'modprobe libcrc32c ; modprobe btrfs' first). For reference, I use a volume group called vgz which will undoubtedly be different to whatever system you might have.

mkdir /tmp/src
mkdir /tmp/dest
lvcreate -L 1G -s -n snap1 /dev/vgz/lvlenny64root
mount /dev/vgz/snap1 /tmp/src
lvcreate -L 12G -n lvlenny64btrfsroot /dev/vgz
mkfs.btrfs /dev/vgz/lvlenny64btrfsroot
mount -t btrfs /dev/vgz/lvlenny64btrfsroot /tmp/dest
cd /tmp/src
find . -print | cpio -dumpv /tmp/dest
cd /
# You'll need to edit the fstab to change the root device and filesystem type. For example:
#    /dev/mapper/vgz-lvlenny64btrfsroot /               btrfs    errors=remount-ro 0       1
vi /tmp/dest/etc/fstab
umount /tmp/dest
lvremove /dev/vgz/snap1
umount /tmp/src

Now we need to convince your initrd to load the btrfs modules (this is debian/ubuntu/whatever specific). I edited /etc/initramfs-tools/modules and added these lines in;

libcrc32c
zlib_deflate
crc32c
btrfs

And then you need to run update-initramfs -u
At this point I had to do some fiddling with /boot/grub/menu.lst. I just made an extra stanza in the file that looked like;

title           BTRFS Debian GNU/Linux, kernel 2.6.31-custom2
root            (hd1,1)
kernel          /vmlinuz-2.6.31-custom2 root=/dev/mapper/vgz-lvlenny64btrfsroot rootfstype=btrfs ro
initrd          /initrd.img-2.6.31-custom2

Obviously don't just copy this for your own setup, as the root line will be different and possibly some other stuff. Admitedly, I am probably missing something with this setup as when I go to boot off my btrfs root partition, I get kernel boot message up to the point where it's mounting the root filesystem, then just sits there for 5 minutes before eventually mounting it ok and continuing on. It's all a bit odd (UPDATE: This long pause is to do with the initramfs fstype and/or vol_id tools not being able to recognise btrfs filesystems. Eventually someone will update those tools, but until then you might want to edit /usr/share/initramfs-tools/scripts/local, hack the get_fstype function and then run update-initramfs-u. I'll put some notes at the bottom of the post).

Anyway, so I got a bootable system with btrfs. The first thing I wanted to try was snapshots, so I duely did something like;

btrfsctl -s snap1 /

When I first did that I thought it would create a snap1 directory under the / (ie. root) of my root fs. It didn't. It actually created a snap1 directory in the directory I was currently in. So within that directory was an intact snapshot of my root filesystem. Of course the first thing you want to do is delete the snapshot ... which in 2.6.31 is a 'future feature'. You can try and rm -rf the snapshot directory ... but I always got some circular file reference errors and it took ages as well. In fact I took a few snapshots as I played around, then left it running over night. In the morning it told me my filesystem was full and the hard drive light was permanantly on. Odd.

Fortunately, the ability to delete snapshots has actually been added to btrfs. You just need an even more recent kernel ... or what I did was to just check out the btrfs kernel tree (I went to here and clicked 'tree' up the top, then 'fs', then 'btrfs' then the 'snapshot' button near the top) and shove it into my 2.6.31 kernel and go through the whole make-kpkg thing again and try booting again (actually I did a few more steps and just deleted my whole btrfs example logical volume and created a new one with no snapshots).

Now with my '2.6.31 and a bit' kernel I can make a snapshot (I have now learned that putting a leading / in front of the snapshot name is probably sensible);

btrfsctl -s /snap1 /

And so if I cd /snap1 I can access my snapshot. But more importantly I can delete it;

btrfsctl -D snap1 /

The arguments to the 'btrfsctl -D' seem to be the snapshot name followed by the directory in which it was made.

I also found this post on root'ing the net that shows how you can mount the snapshot using the mount command (ie. in addition to the snapshot appearing as a subdirectory);

mount -o subvol=snap1 /dev/vgz/lvlenny64btrfsroot /mnt

Most stuff on btrfs indicates that subvols and snapshots are treated the same. If you create a new snapshot you get a copy of what you snapshotted. Creating a new subvol gives you an empty directory which I assume can be controlled separately, but uses the same pool of disk resources of my main btrfs filesystem. I'm not entirely sure. There is an option to the btrfsctl command which can resize a 'something' but I'm not too sure exactly what it does. I tried resizing down and df doesn't reflect any difference, but doing a btrfs-show did show some change. At the moment there aren't really any commands to list snapshots or show usage. There is a btrfs-debug-tree command that spits out reams of info ... and does mention my snapshots deep within it.

btrfs can apparently do RAID1, RAID0 and RAID10. I like that redundancy is a feature of the filesystem. I've never really understood why LVM2 for linux has no redundancy features (and the mirroring capability it does have is not very useful). I haven't tried any of the btrfs RAID features yet.

So that's all I've looked at so far. One of the key features of ZFS that I'd really like to see in btrfs is 'rollback to snapshot'. This is such an incredibly useful feature in a fast paced IT environment. Given that btrfs is a copy-on-write filesystem, I am hoping they put rollback in at some point.

Honestly I haven't tested it enough to determine how stable it is. It seems fine so far, but this is not a server type machine.

UPDATE: re getting rid of the long pause on boot.
I edited /usr/share/initramfs-tools/scripts/local, and added the extra lines shown below. Be warned, this is an awful 'hack' that assumes that any filesystem type that is not recognised is 'btrfs'. Once you've updated the file you'll need to run update-initramfs -u

get_fstype ()
{
        local FS FSTYPE FSSIZE RET
        FS="${1}"

        # vol_id has a more complete list of file systems,
        # but fstype is more robust
        eval $(fstype "${FS}" 2> /dev/null)
        if [ "$FSTYPE" = "unknown" ] && [ -x /lib/udev/vol_id ]; then
                FSTYPE=$(/lib/udev/vol_id -t "${FS}" 2> /dev/null)
        fi
        RET=$?

        if [ -z "${FSTYPE}" ]; then
                FSTYPE="unknown"
        fi

# Hack for my btrfs root. Terrible hack
        if [ "${FSTYPE}" = unknown ];then
                FSTYPE=btrfs
                RET=0
        fi
# End of my hack

        echo "${FSTYPE}"
        return ${RET}
}

There are quite a lot of ways to get a remote graphical desktop when connecting to a unix system. X windows itself has always been a network based protocol, so if you had an X terminal or a PC running an X server, or even another unix system, you could always connect in that way. However, in more modern times with locked down Windows desktops in corporations, VNC has become very popular. Since something like the tightvnc client is just a standalone executable (and doesnt require installation), it’s often the simplest way of getting a graphical desktop going from a Windows PC. Another advantage is that VNC just uses a single tcp port connection, so it’s often a simple port forward and you can connect.

I’ve used VNC for years, and it is indeed handy, but it’s never been a speed demon. Sure there are various compressions and encodings you can choose, but it’s never that great (however I have discovered tigervnc in the course of researching this article, and it does look very promising). I often end up running vnc in bgr233 (8 bit) mode in order to make the desktop more responsive, but at the expense of screwing up most of the colours on the screen. Of course, if you look at other remote screen technologies they tend to be a lot better. Windows terminal server (RDP) client access always seems quite fluid to me. Citrix always seems pretty good too, NoMachine is very good, but I always find it difficult to set up. The Sunray protocol is very good too.

So recently, I started looking at xrdp. It’s been around for ages, and almost looks like a dead project at times … but if you look closely at the mailing list … it is still ticking over. I always thought it was a RDP server for linux. Well it is sort of. Regardless of what it is, it provides you with a very responsive remote desktop experience.

So xrdp seems to do a few things. The main uses are as an authentication gateway to VNC or other RDP server. So if you run xrdp on your system it starts listening on port 3389 like a normal RDP server. You can connect to it with rdesktop or the Windows terminal services client. Basically you see a login screen. There’s a dropdown (yep the square grey box is a dropdown) that allows you to select different connection profiles. You define these yourself in /etc/xrdp/xrdp.ini. You always get a few examples in this file but they’re never really explained anywhere.

All the examples have a [xrdp] header. I’m not sure if they have to be [xrdp1], [xrdp2] etc. Below is an example of a profile that connects to a VNC server running on :1 (port 5901). name is what you see in the drop down. lib = libvnc.so means that you want xrdp to connect to a VNC server and then convert it back and forth between VNC and RDP. You need to start up your VNC server before hand though. You may think “Why not just use a VNC viewer and connect to the VNC server directly?”. Responsiveness. I find that accessing a typical linux desktop via this RDP to VNC method is actually a fair bit better than just connecting via VNC (and this is just on a local LAN). The other params are reasonably obvious. username=na means to not prompt for a username. password, ip and port are reasonably obvious. If they are set to ‘ask’ then you get to fill them in at the login screen. Obviously you can make this connect to any VNC server by using password=ask, ip=ask, port=ask

[xrdp1]
name=VNC5901
lib=libvnc.so
username=na
password=ask
ip=127.0.0.1
port=5901

This next example is more interesting;

[xrdp4]
name=sesman-any
lib=libvnc.so
ip=ask
port=-1
username=ask
password=ask

It still uses libvnc.so so it’s obviously converting between RDP and VNC still. The interesting bit is the username=ask. Most VNC servers don’t take a username. What this profile does is to prompt for a username and password and ip. As part of the xrdp startup it always starts up a daemon called sesman. This is like a login authenication daemon which can spawn a command once authenticated. By default sesman listens on port 3350, so xrdp takes your user/pass and connects to the ip you specified on port 3350. The sesman on that port should respond and authenticate the user (typically the ip is always 127.0.0.1). Then the cool bit is that it starts up Xvnc as your user on the ‘next available DISPLAY setting above :10′, and assuming the Xvnc startup works, xrdp starts the whole RDP to VNC conversion. So you no longer need to start up Xvnc beforehand. Another good thing about this method is that if you just exit your RDP client (ie. don’t choose to logout), then the Xvnc session keeps running in the background, and if you RDP in again later it’s all still working where you left off. Conversely, if you choose to ‘log out’ then the Xvnc process and any windows you had open ends as you log out.

But there are more lib= types you can use. Look at this example from xrdp.ini that uses lib=librdp.so

[xrdp5]
name=windows
lib=librdp.so
ip=ask
port=3389

librdp.so allows you to sort of proxy through your xrdp server to another RDP server (eg. a windows PC)

Then there’s this sesman-X11rdp example;

[xrdp6]
name=sesman-X11rdp
lib=libxup.so
username=ask
password=ask
ip=127.0.0.1
port=-1

For a long time I never really understood what this one did. In fact in 99% of all xrdp installs out there it does absolutely nothing. It asks for your username so that means that sesman thing is involved. So it will authenticate your user/pass with sesman, but instead of starting Xvnc, it starts X11rdp. Chances are you do not have X11rdp. Most linux distros do not include it as part of their ‘xrdp’ packages, so what is it?

X11rdp is a real X server that instead of thinking its talking to say an Nvidia or ATI hardware video card, it is talking to an abstracted ‘RDP’ video card. Similarly the mouse and keyboard are abstracted to come down the RDP protocol connection. So it still talks the X protocol, so that typical X client programs (eg. xterm) still think they’re talking to an X server, but ‘out the other side’ it’s all RDP traffic. I don’t think it talks RDP over a network connection though (not sure). I think the role of libxup.so is to provide the ‘link’ to it.

So where do you get X11rdp? I couldn’t find many references to it on the net but came across this post that mentions that there is an svn repository containing a modified Xorg 7.1 server that when compiled produces the X11rdp binary. Compiling the old Xorg drivers takes a long long time, but I had success following those notes. So here’s what I did;

cd some_temp_dir_with_plenty_of_space
svn co svn://server1.xrdp.org/srv/svn/repos/main/x11rdp_xorg71
cd x11rdp_xord71
mkdir /opt/X11rdp
sh buildx.sh /opt/X11rdp
# and now wait a long long time

That post on linuxquestions.org mentioned some changes to font directory paths inside the buildx.sh script. I didn’t have to change it at all the first time I tried compiling. I used a Debian Lenny 64 bit host to compile. However, I wanted to put this on a Centos 5.3 32 bit system as well and it wasn’t so good. basically there are a couple of changes to the buildx.sh script. First it compiled libfreetype as a 64 bit binary, so I had to add in the CC=”gcc -m32″ bit below

# freetype
if ! test -f $PCFILEDIR/freetype2.pc
then
  cd freetype-2.1.10
  CC="gcc -m32" ./configure --prefix=$PREFIXDIR
  if ! test $? -eq 0
  then
    echo "error freetype"

And the font path in Centos is a bit different to what the script was expecting, so I had to do the following (adding in the last elif bit)

# make a symbolic link to your local font directory
if ! test -d $X11RDPBASE/lib/X11/fonts
then
  if test -d /usr/share/fonts/X11
  then
    ln -s /usr/share/fonts/X11 $X11RDPBASE/lib/X11/fonts
  elif test -d /usr/X11R6/lib/X11/fonts
  then
    ln -s /usr/X11R6/lib/X11/fonts $X11RDPBASE/lib/X11/fonts
  elif test -d /usr/share/X11/fonts
  then
    ln -s /usr/share/X11/fonts $X11RDPBASE/lib/X11/fonts
  fi
fi

I also compiled for a Debian 32 bit build and had to put ‘CC=”gcc -m32″;export CC’ at the very top of the build script (but that might be because of the virtual machine I was compiling in).

So once you have X11rdp compiled, make sure it’s in your path (perhaps create a symlink from /usr/bin/X11rdp to wherever you have it).

So for me xrdp plus X11rdp works great, but you could never copy and paste between your local environment and your remote environment … until now. As well as the stable releases of xrdp, there is also a CVS repository, and one of the more recent commits (sept 2009) is to add clipboard support (thank you Jay). To get and compile the cvs version you need to check it out.

mkdir workdir
cd workdir
cvs -d:pserver:anonymous@xrdp.cvs.sourceforge.net:/cvsroot/xrdp checkout xrdp
cd xrdp
./bootstrap
./configure
make install

You might have to do some fiddling with various paths to get it all going (most distro’s seem to run xrdp as a non-root user. If you run the cvs version this way it will have trouble writing the pid file when xrdp starts … unless you change the path to the pid file … or do a ./configure –prefix=/usr/local/xrdp … or similar. The best way is to just use strace when launching the binaries to see what’s going on)

You run xrdp then xrdp-sesman (which used to be called just ‘sesman’). The easy way is to run both as root, though you might want to invest the time to get xrdp to run as a non-root user. For example;

su - xrdp -c /usr/local/sbin/xrdp
/usr/local/sbin/xrdp-sesman

The clipboard support just ‘works’ for the most part. I had a few issues with using the Windows XP mstsc client, but rdesktop and the Windows 7 mstsc seem to work fine (UPDATE: It’s only the XP SP2 mstsc that I can’t get to work with the clipboard. The mstsc out of XP SP3 works fine … and in fact works fine if you copy it to a SP2 machine)

(Note: This post was initially written when ESXi 4.0 was available. As of late 2010, ESXi 4.1 has been released, and it does actually include a sky2 driver that may or may not work with various Marvell LAN chipsets. The post is still relevant (especially the comments)  if your particular Marvell chipset does not work with the sky2 driver in ESXI 4.1. Also, the post is relevant if you’re interested in porting other network drivers to ESXi)

Well, after somehow getting my Marvell LAN card working with ESXi 3.5u4 (and u3) I thought I’d have a look at  ESXi 4. Again I somehow got it to go. I’m not too sure how good it works, but it works well enough for me at home. If you can’t be bothered reading about me going on and on and on and on about how to compile it, then just scroll to the bottom of the post. The download for the source includes a precompiled module (NB: As per the post about ESXi 3.5, this is all about getting a 88E8053 chipset Marvell LAN working).

ESX/ESXi 4 is quite different from 3.5. The build chain is similar to 64 bit Redhat/Centos 5.2, so I ended up installing a x86_64 Centos 5.3 inside a vmware fusion machine to do my dev work. I just made sure I installed all the dev stuff.  Then I downloaded the VMware-esx-public-source-4.0-162945.tar.gz from VMware’s open source page. It’s a much bigger file (590MB) than the file for 3.5. When you extract the file you end up with a lot of rpm files plus a vmkdrivers-gpl.tgz file. I did the following to extract it all on my test machine;

cd ~
mkdir vmware-oss
cd vmware-oss
tar xvzf ~/VMware-esx-public-source-4.0-162945.tar.gz
mkdir drivers
cd drivers
tar xvzf ../vmkdrivers-gpl.tgz

One of the rpm files included is a kernel source rpm. I’m not exactly sure what it is relevant to ESX, but I installed it anyway for reference. I found I needed the qt-devel and gtk2-devel packages first;

cd ~
cd vmware-oss
yum install qt-devel
yum install gtk2-devel
rpm -iv kernel-sourcecode-400.2.6.18-128.1.1.0.4.159770.x86_64.rpm

I’m pretty sure you don’t need the kernel source to build the drivers, but I kept it handy for reference anyway.

You can try doing a test build of the drivers now. This will build all the drivers built in to ESX/ESXi.

cd ~/vmware-oss/drivers
./build-vmkdrivers.sh

You’ll probably get a few warnings, but it should complete. If you do a find down the ‘bora’ directory you should see a bunch of .o files corresponding to the kernel modules (look under the ‘bora/build/scons/build’ directory).

OK, my approach was to look at the build-vmkdrivers.sh script and basically look at what was done to compile one network driver (I used the forcedeth driver as a reference) and just make a reduced script for my sky2 driver.  As for the source to base the sky2 driver on, instead of using a driver from 2.4.37 like I did with the 3.5 version of the network driver, this time I ended up using the sky2 source from 2.6.26 (or the debian lenny incantation of it). I did originally use the sky2 driver from the kernel-sourcecode…2.6.18…rpm file, but on closer inspection of the tg3 driver that ESX uses, I noticed it actually comes from a 2.6.24.1 kernel (or thereabouts) … so I thought I may as well use a more modern reference source. I had a few minor hiccups trying to get it to compile, but in the end I just had the following shoved into the top of my sky2.c file;

/* Stuff for ESX compile */
#define upper_32_bits(n) ((u32)(((n) >> 16) >> 16))
#define csum_offset csum
#define bool int
#define PTR_ALIGN(p, a)         ((typeof(p))ALIGN((unsigned long)(p), (a)))

u32 bitreverse(u32 x)
{
        x = (x >> 16) | (x << 16);
        x = (x >> 8 & 0x00ff00ff) | (x << 8 & 0xff00ff00);
        x = (x >> 4 & 0x0f0f0f0f) | (x << 4 & 0xf0f0f0f0);
        x = (x >> 2 & 0x33333333) | (x << 2 & 0xcccccccc);
        x = (x >> 1 & 0x55555555) | (x << 1 & 0xaaaaaaaa);
        return x;
}

The define’s are to remedy compilation errors, and the bitreverse is to satisfy an undefined symbol problem. Note that the undefined symbols errors end up in /var/log/messages on your ESXi 4 box now (unlike 3.5).

So in the end to compile my driver I did a;

./build-sky2.sh

You get a couple of warnings, but if you do a ‘find . -name sky2.o’ you should end up with two sky2.o files. There is a DEBUG and DASHG variable defined at the top of the build script. If you uncomment these it’ll build a lot of debug stuff into the modules.

If you get some errors, its probably because some directories are missing in the build path, so make them first;

mkdir -p bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/SUBDIRS/vmkdrivers/src26/drivers/net/sky2
mkdir -p bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/SUBDIRS/vmkdrivers/src26/common/

Now, again I used a USB stick with ESXi 4 (build 171294 in my case). To install it, I loopback mounted the VMware ESXi 4 iso file file, extracted the image.tgz file to a temp directory, bunzip2′d the big dd image file, then dd’d it to the whole USB stick (NB: There are some details about how to do this in linux on my other post re ESXi 3.5. See link at the top of the post)

A difference this time is that I didn’t have a simple.map file all pre-prepared to go into the oem.tgz file. I thought the easiest way to get it would be to just boot ESXi and let it fail when it tries to configure a network device, then somehow copy the simple.map file off. So I did this. ESXi 4 merrily boots and eventually you see the dreaded ‘lvmdriver failed’ message. It looks like ESXi is broken at that point, but just type the word ‘unsupported’ and you get a password prompt, and just hit ENTER to get a prompt (You might need to hit alt-f1 first before typing ‘unsupported’)

Because networking is not working, we’ll just copy the simple.map to the Hypervisor1 partition;

cp /etc/vmware/simple.map /vmfs/volumes/Hypervisor1

I just did a ‘sync’ and held in  the power switch ( perhaps type ‘reboot’ if you feel like being more careful). Now get the USB stick to appear as a USB device in your development VM (your centos 5.x environment), and mount partition 5 (or the Hypervisor1 partition) off the USB drive, and you should see an oem.tgz file as well as the simple.map file. You need to make a directory structure up for the new oem.tgz file we’ll be creating;

cd ~
mkdir vmtest
cd vmtest
mkdir -p etc/vmware
mkdir -p usr/lib/vmware/vmkmod

Copy the simple.map off the USB drive into etc/vmware directory in our tree structure. eg.

cp /mnt/simple.map ~/vmtest/etc/vmware

And edit the simple.map so that it includes the PCI ids for your Marvell card. Mine is 11ab:4362 so I added in the bolded line below, but yours could likely be different. If you’re not sure, you could boot ESXi off the USB stick again, do the ‘unsupported’ thing to get a prompt and type lspci -v

1166:0410 0000:0000 storage sata_svw.o
1166:0411 0000:0000 storage sata_svw.o
11ab:4362 0000:0000 network sky2.o
14e4:1600 0000:0000 network tg3.o

Now copy in the sky2.o file that we compiled earlier. The modules are in a different directory compared to 3.5 (NB: the compilation process produces two sky2.o files, so make sure you grab the one shown below)

cd ~/vmware-oss/drivers
cp ./bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/sky2.o ~/vmtest/usr/lib/vmware/vmkmod

Now tar it up, and copy it to the USB stick that should be still mounted;

cd ~/vmtest
tar cvzf ~/oem.tgz *
cp ../oem.tgz /mnt

Unmount the USB stick

umount /mnt

Now try booting again. Hopefully you should see a ‘loading sky2′ flash up early in the boot … and it should eventually get to the usual ESX status screen showing the current mgmt IP address. Basically if you don’t see the ‘lvmdriver load failed’ then there’s a good chance it’s working.

And yes here is the sky2-for-esxi4-0.01.tar.gz download. It includes the build script, the modified source, plus directory tree for creating the oem.tgz file including a pre-compiled copy of the module. If you can’t be bothered compiling, you can just extract this file, cd to the vmtest directory and create the oem.tgz file as per the earlier notes.

UPDATE: (2010/02/08) There is also now a driver for the Marvell 88E8001 LAN chipset (see the comments discussion below). This uses the skge driver, not the sky2 driver mentioned above. I don’t own a 88E8001, so thank you to samarium for helping out re the unresolved symbols. Please comment below if you’ve tried the skge driver and it works. I have a new tarball sky2-and-skge-for-esxi4-0.02.tar.gz containing both the sky2 and skge driver

UPDATE: (2010/12/24). xieliwei has a driver that should support the 88E8057 on ESXi 4.1 (as per the latest comments, it seems a bit iffy on 4.0). Anyone else who has this chipset, if you could try the driver and post more information that would be great. The 88E8057 code is here as a local copy ; sky2-1.22-for-esxi4-88e8057-r1-xieliwei.tar.gz or on mediafire; sky2-1.22-for-esxi4-88e8057-r1-xieliwei.tar.gz . Also there’s another version of the driver that will produce copious debug information (only of use if you’re having problems); http://www.mediafire.com/file/m836gce3r3b7ygi/sky2-1.22-for-esxi4-88e8057-r1-oem-debug-xieliwei.tgz