Reverse Engineering a vDisk (VHD)

So you’ve got this awesome Golden Image that has everything on it you could possibly want. You can make that puppy boot to bare metal and to ESX and even to Hyper-V if you feel like it.

But then Citrix comes out with a new Provisioning Server update that requires you to unprovision or Reverse Engineer that vhd to do updates. There are a handful of reasons why this might be unnecessary:

  • New versions of PVS require Target Device software updates
  • New NIC
  • New NIC driver update that needs to be applied

Any one of those could BSOD your server if you try to do them to a provisioned vDisk so you need to RE them to do the update.

        1. Make a copy of your vhd and boot it into Private Mode on a server
        2. Open Disk Management and ensure that the local hard disk is showing online. It should show up as Disk 0
        3. Format the local hard disk (if you’ve moved your page file or spooler or anything to this disk, you’ll need to move those off first)
        4. Go to a command prompt and run C:Program Files (x86)CitrixXenAppPrepXenAppPrep.exe /pvs
        5. Click on Start —> All Programs —> Citrix —> XenConvert
        6. Select “From Volume to Volume”
        7. Select Source Volume as D: and destination volume as C:
        8. Select Next and start the Conversion. This will take several hours.
        9. After Conversion reboot and go into the BIOS and select to Boot from Local Disk
        10. Go to control panel and uninstall the “Citrix Provisioning Services Target Device x64”
        11. Reboot
        12. After the reboot make any changes you need to make (NIC updates, adding/removing NICs, etc)
        13. Reboot again
        14. Reinstall the Target Device software from the PVS disk
        15. Go to your PVS server and open the PVS Console
        16. Go to properties of the target device and select “Boot from: Hard Disk
        17. Create a new vDisk in the PVS Console and assign it to the target device
        18. Reboot the target device and go into the BIOS. Change the boot order to boot from network.
        19. Open Disk Management and ensure the local hard disk is showing online. It should show up as Disk 1. You should also see a Disk 1
        20. Format Disk 1
        21. Go to a command prompt and run C:Program Files (x86)CitrixXenAppPrepXenAppPrep.exe /pvs

      1. Run XenConvert again and image from the local hard drive (C:) to the VHD (D:)
      2. This will take several hours.
      3. When it’s done, shut down the target device.
      4. Go back into the PVS console and change the properties of the Target Device to boot to vDisk.

You’re done! At this point you’ve still got your old VHD and now your new one with all the updates. You’ll also have everything on the local hard disk on that target device so you may want to go back after you boot it thru PVS and re-format that disk so that you don’t have any booting accidents.


Provisioning Server 5.6 and cross-platform VHD’s, Part 2

So you’ve decided (or been told by management) that your VHD’s need to be able to work with both bare-metal and ESX (and Hyper-V). There are many different reasons why this is a valid thing to do. The major ones being portability between different hardware and the ability to use some kind of replication software (DFS-R for example) to copy your VHD to another site and automatically bring servers up there.

If you read Part 1 of this post you now know how to get your image to work between bare-metal and Hyper-V. If Hyper-V is necessary in your environment you can really do that piece before or after the process I’m about to explain. It’s kind of adjunct process.

The process for making your VHD/vDisk work with both bare-metal and ESX is mostly straightforward. It’s not the simplest procedure, but it’s the one I’ve found that consistently works.

The first thing you want to do is go to and download the V2V Converter. You can use XenCovnert to go P2V or V2P or use the VMWare Converter to do the same, but those utilities all insert their own little functionality which manages to break your VHD. the V2V converter does a block level conversion from an ESX .vmdk to a standard .vhd and vice-versa.

You’ll want to install this utility on the Citrix Provisioning Server, or wherever you’re hosting your vDisks.

  1. Boot your vDisk into Private Mode on one of your bare metal servers
  2. Copy the PVS Target Device software to a local temp directory. Also copy any other software/drivers you might need. When you convert the .vhd to a .vmdk you will NOT have any network access. (You could also mount your .vhd on the PVS server directly and copy the files into the temp location.)
  3. Shut down the bare-metal server and physically copy the .vhd file to a temp location on the PVS server.
  4. Run Starwinds V2V Image Converter.
  5. Browse to the source VHD and click Next
  6. Select VMWare Growable Image and click Next (assuming you thin-provision)
  7. IMPORTANT: Select IDE disk, even if it’s not.
  8. Select the name/location of the output .vmdk
  9. Let the conversion go…
  10. Install the vShphere client on your PVS server. Open it and log into vCenter.
  11. Browse to your datastores and create a Folder for your temp VM.
  12. Upload the .vmdk to this folder
  13. Create a new virtual machine. Give it 2 processors, however much memory you want, specify the E1000 NIC (super important for PVS), then UNCHECK the “Connect at Power on” box. You don’t want this bad boy on the network.
  14. When you get to the Virtual Disk screen select “use an existing” and browse to your uploaded .vmdk in the store.
  15. Power on the VM
  16. Log into the console with cached creds.
  17. Install the VMWare tools to get the appropriate drivers installed.
  18. Go to Control Panel and uninstall the Citrix Provisioning Services Target Device x64 software.
  19. Reboot.
  20. Log back in and reinstall the Target Device Software from your temp location. You remembered to copy it in, right?
  21. When it’s done, make sure all your NICs are checked and click OK. You should see at least one that is inactive.
  22. Reboot
  23. Power off the VM
  24. Repeat the above steps to download the .vmdk from the datastore back to your local temp location.
  25. Open the StarWind V2V Image Converter.
  26. Select the new .vmdk file (you’ll probably see temp.vmkd and temp-s001.vmkd, select the temp.vmdk)
  27. Select the destination as “MS Virtual PC growable image” and click Next. Again, this assumes you’re thin-provisioning.
  28. Select the output filename/location.
  29. When done, just copy the .vhd back into the PVS datastore and import it as usual.

As convoluted as that sounds, that’s really all you need to do. Your image will now be able to boot between both ESX and bare-metal. You’ll of course want to clean up the image a bit. Uninstall any hardware agents (HP SIM, etc.). And you’ll get some errors in the logs when you boot that it can’t find so-and-so hardware, but for the most part you should be set.

I’ve got my images booting between bare-metal (HP GL460c G6), Hyper-V, and ESXi 4. It’s pretty slick and took months to figure out the right order for everything but it now runs like a dream.

Of course if you followed the Citrix best-practices you were already using ESX or XenServer (same process, btw), so you probably don’t have this problem. But if you’re like most people don’t RTFM you’ll hit this pickle at some point.

Provisioning Server 5.6 and cross-platform VHD’s, Part 1

So if you’re like anyone, you actually buy new hardware. Today you’ve got a BL460c G5, tomorrow you can only buy G6’s. Or G7’s. You’ve also got an ESX environment and maybe even a Hyper-V server stashed somewhere. The beauty of Citrix PVS and streaming your VHD’s is that you can have bare metal just sitting there… sitting there doing nothing, and you can PXE boot it and be on your way.

You also like to stream your images to your VM’s, cause hey… You can virtualize your hardware, why not virtualize the OS you put on it?

And maybe you’ve got that Hyper-V server sitting over there, off to the side, and you want to stream your images to it as well, cause, you know, you can.

Soon enough you’ll be like me, and have 12 different images covering 3 different environments, and you realize what a total PITA is it to manage all these darn things. Because officially Citrix says they don’t support moving that image to different hardware and if you try it you get the dreaded BSOD.

But if you’re me, you don’t accept that answer…

The only way Citrix supports moving your image from one set of hardware to another is to use XenConvert and P2V it back to bare metal, then take the drives out and slap them in the other hardware. I’ve done it, and it’s not pretty, but it DOES work.

I found a relatively cool way online to make my G6 image work on Hyper-V. It was actually pretty straightforward:

  1. Do your normal build process for your image on the G6 and set it to Private mode.
  2. Copy the vhd from your PVS server over the network to your Hyper-V machine. Create a new VM with multiple processors, an appropriate amount of ram, and attach the VHD to it as it’s hard drive.
  3. Very important.. remove the default NIC and add the “Legacy NIC”
  4. Boot your VM, log into the machine and let it install all the stuff it needs to work.
  5. Uninstall the PVS Target Device Software, reboot, reinstall it, then when it comes up make sure to tell it to use all of the NICs, even the ones it says are unavailable.
  6. You see, the problem with PVS is that it loads all the nic’s it’s bound to when it starts the GUI boot. If it doesn’t load the actual device (not just the driver), then you get your BSOD.
  7. So after you reinstall the software, reboot again.
  8. Run XenAppPrep.exe /pvs from the command line and shut down your machine.
  9. Go back to the file system and copy the VHD back over to the PVS data store, add it thru the console and you should be good to go to boot the image from either your original physical or to Hyper-V.

Great! You’re done, right?

Well, not so much. Wouldn’t you rather have your image be hardware-independent? Wouldn’t you like to boot it thru ESX?

This process doesn’t remotely work for ESX.

Oh, and didn’t I mention? This only works with 2008. If you want 2003 or anything earlier, good luck. There aren’t any drivers for the legacy NIC in Window 2003.

If you want hardware independence, read on for Part 2.

Basic Troubleshooting

I run into this all the time. You’ve got a complex application that runs across a multitude of servers and users are randomly reporting errors. How do you figure out what’s producing the error?

Let’s run through a scenario, then I’ll give you my process on it:

Users launch the application through Citrix. The app launches an IE session which runs on 5 different Citrix servers. All Citrix servers are (supposedly) built the same. Even though the application is purely web-based there are Java controls and vendor-proprietary controls that need to run on the machine running IE. Rather than have that all installed on each machine you put it on Citrix.

The IE launches using a Virtual IP address that points to a Big IP (or any kind of load balancer). Behind the load balancer are 10 web servers (again, supposedly built the same). Then behind the web servers are application servers, SQL servers, file servers, etc.

So when a random user calls into the Helpdesk and reports that when they launch the application and log in, they get an error. They attach a screenshot to the ticket and it’s definitely being produced by the web site. In this case it was an assembly error.

So what do you do?

1)    Determine if this error is happening to multiple users. If it’s just 1 user it’s probably a profile issue or a local caching issue. In that case just delete their roaming profile and their local profile on their machine to see if that fixes it.
2)    If multiple users are having the problem, as in this case, time to go on to the next step.
3)    Is it happening to all users all the time? In this case, it isn’t. It’s “random”.
4)    This, unfortunately, is where the tediousness comes in. Your error is being produced in IE, but is it a web site issue or a Citrix issue? Or a backend issue?
5)    You can quickly eliminate it being a backend issue, as it works for some people. If it wasn’t working for everybody that would be a pretty good indicator you got major problems going on.
6)    So the first thing to try is to see if you can log into all 5 of your Citrix servers. Usually I have a test app (Notepad.exe) published against each server just for this purpose. You can also try RDP into each machine, but that doesn’t tell you whether Citrix is working or not.
7)    Assuming your test app is working all right, go RDP into each Citrix server. You know Citrix itself is working all right.
8)    Open iE on your Citrix server. Browse to the login page of your first web server (not the load balanced address) and test login. If it works go on to the next one. Do this against all 10 web servers. If they all work, then you’ve got a problem with your load balancer. If none of them or 1 or 2 produce an error, you now know that it’s a problem with a web server. If the problem only happens on 1 Citrix server against all the web servers, it’s probably a problem with your web controls on that box.
9)    Confused yet? You know where you’re going with this. Test all 10 web servers on all 5 Citrix servers. See if your problem is consistent or if it travels at all.
10)    In this case all Citrix servers had a problem against web06. So we knew we had a problem with web06. Quick fix is to take it out of the load balancer and bypass the problem for the users.

The ultimate fix is to figure out what the heck is wrong with your web server. In this case it was that the controls weren’t updated correctly the last time they were updated. Problem solved.

Your takeway from this is that you need to know how your apps map out and the total flow of data. Then you just need to chip away at the problem, eliminating options as you go, until you figure it out. In the above case if we’d just kept trying the load balanced VIP we’d have gotten nowhere from the get-go.

SEP11 and Citrix PVS

So in my inherited Citrix Provisioning Server (PVS) 5.6 environment with SEP11, we recently noticed that none of the devices were showing up correctly in the SEP admin console (or at all), plus the Symantec Management Client service (which is set to automatic) wasn’t started. You could go into the services mmc (thru Server Manager or services.msc from the Run, which is my fave) and start it up manually and the device would appear in the console and everything was fine.

Reboot and you’d be back to where you started, which is typical with PVS disks in Standard Mode.

You see this in the Event Log:

Symantec Management Client service depends on a service in a group which starts later.


Detected circular dependencies auto-starting services. Check the service dependency tree.

Hey, that seems easy enough, right?

So after some googling and playing around with it I made a copy of my vhd, set it to Private mode and booted it up. Once up I went into the registry and browsed to


By default the value is set to NDIS. If you set it to NDIS2 everything starts working. So just run your Device Optimization Wizard, run your shutdown script (or xenappprep /pvs), shut down, then set your image back to Standard Mode and there you go.