Windows Server “8” Preview

If you’ve read my previous post below – and you ARE following me religiously, right? – then you know that I really don’t like to read the manual (RTFM!). That being said, I recently downloaded the Windows Server “8” beta and tried to throw it up on my ESXi 5.0 servers. And they blue-screened.

Thankfully it had a frowny-face, so at least I felt better about it.

After some googling/digging I discovered that Win Server 8 isn’t really supported on ESX yet. Some people report success, some don’t. Supposedly if you update to 5.0 Patch 2 it might work. Or so the articles say.

Unfortunately for me, I can’t really take downtime right now to really test it. So on to another solution…

VMWare Workstation 8. Supports it like a charm. The key is to NOT specify OS customization during the creation of the VM. Just tell it you’ll install the OS later and then mount the ISO and do your thing.

Of course then you’ve got the problem of sharing resources on your machine with your VM, so hopefully you have a beefy machine (unlike me). I allocated 2 CPU’s and 1 GB of RAM to the VM. It’s a little clunky, but it’s passable.

Stay tuned as I start testing Win 8. I have a few items of interest I’m looking at, but will also give my general comments as I run along things. I’m mainly looking at the new multi-tenancy features of AD, CIFS2 file sharing, and just general interface changes. I’ve never been a fan of Hyper-V, and am not sure I can run it within a VM, but I might look at that, too, since everyone says it’ll do away with Citrix and I happen to love all things Citrix.


Reverse Engineering a vDisk (VHD)

So you’ve got this awesome Golden Image that has everything on it you could possibly want. You can make that puppy boot to bare metal and to ESX and even to Hyper-V if you feel like it.

But then Citrix comes out with a new Provisioning Server update that requires you to unprovision or Reverse Engineer that vhd to do updates. There are a handful of reasons why this might be unnecessary:

  • New versions of PVS require Target Device software updates
  • New NIC
  • New NIC driver update that needs to be applied

Any one of those could BSOD your server if you try to do them to a provisioned vDisk so you need to RE them to do the update.

        1. Make a copy of your vhd and boot it into Private Mode on a server
        2. Open Disk Management and ensure that the local hard disk is showing online. It should show up as Disk 0
        3. Format the local hard disk (if you’ve moved your page file or spooler or anything to this disk, you’ll need to move those off first)
        4. Go to a command prompt and run C:Program Files (x86)CitrixXenAppPrepXenAppPrep.exe /pvs
        5. Click on Start —> All Programs —> Citrix —> XenConvert
        6. Select “From Volume to Volume”
        7. Select Source Volume as D: and destination volume as C:
        8. Select Next and start the Conversion. This will take several hours.
        9. After Conversion reboot and go into the BIOS and select to Boot from Local Disk
        10. Go to control panel and uninstall the “Citrix Provisioning Services Target Device x64”
        11. Reboot
        12. After the reboot make any changes you need to make (NIC updates, adding/removing NICs, etc)
        13. Reboot again
        14. Reinstall the Target Device software from the PVS disk
        15. Go to your PVS server and open the PVS Console
        16. Go to properties of the target device and select “Boot from: Hard Disk
        17. Create a new vDisk in the PVS Console and assign it to the target device
        18. Reboot the target device and go into the BIOS. Change the boot order to boot from network.
        19. Open Disk Management and ensure the local hard disk is showing online. It should show up as Disk 1. You should also see a Disk 1
        20. Format Disk 1
        21. Go to a command prompt and run C:Program Files (x86)CitrixXenAppPrepXenAppPrep.exe /pvs

      1. Run XenConvert again and image from the local hard drive (C:) to the VHD (D:)
      2. This will take several hours.
      3. When it’s done, shut down the target device.
      4. Go back into the PVS console and change the properties of the Target Device to boot to vDisk.

You’re done! At this point you’ve still got your old VHD and now your new one with all the updates. You’ll also have everything on the local hard disk on that target device so you may want to go back after you boot it thru PVS and re-format that disk so that you don’t have any booting accidents.

UPM and Mandatory Profiles

So I woke up one day last week to a critical ticket from the Helpdesk. We were getting “tons of calls” from users of one of our Citrix applications that they couldn’t launch it. They’d log in through the WI and when they clicked to launch the app were getting the error: “The Group Policy Client Services failed to Log On. Access is Denied.”

The tickets were coming in like a flood but when I went to go check the application in the Citrix AMC I could see we already had about 200 users connected (normally it’s like 250). So, okay, the application itself and the Citrix servers were okay.

Now let me backtrack for a second and give a quick run down of our environment. We’ve got well over 50K users spread across about 35 Windows AD domains, all in one forest. Very few users have access to all of the domains and until the day after this incident happened I wasn’t one of them. But like any user I’ve got read access to all objects so after about a half hour or so of troubleshooting on the server side and not finding anything I went back to the users.

All of the users were in 2 different domains, both of which happened to be in the same physical location and their domain admins were in the process of migrating users from 1 domain into the other. So, hey, commonality!

My Citrix servers are using UPM 3.2.2 for Profile Management and we save the profiles to a file server at the main site and use DFS to replicate them to our secondary site for DR. Not much is allowed to be saved into their profiles, but there are a few tiny things that are needed for the application.

A google search turned up a few things to look at on the user accounts so I started looking at their Terminal Profile tab. Now if you know UPM at all you know it doesn’t play well with any other kind of profile management. We’ve got some GPO settings in place to try and disable the use of terminal services profiles, but there’s no real way to fully disable the use of them. We’ve had conflicts before, so it’s a process.

In this case all the users had the exact same settings on their Terminal Profile tab. It looked weird and after a few seconds of looking at them it became obvious. It was pointing to serversharedirectory. And that was it. You’d think it should have been pointing to something like servershare%username% or some such. So this meant those users had Mandatory user profile configured. Do you know the one thing that UPM doesn’t work at all with?

Yep, Mandatory user profiles.

After that, the fix was easy. We had to remove the settings from their account, replicate, delete their UPM profile on the share, then they were fine.

Local support said that those were “old” settings and no longer used, so we could delete them without any problem. We still haven’t answered the obvious question: If we changed nothing, and local support changed nothing on their domain, then why did this setting suddenly matter or make a difference?

The best we can figure is some policy was enabled or disabled on either our domain or theirs that enabled that setting to be used, but I’ll be darned if we can figure out what.

Provisioning Server 5.6 and cross-platform VHD’s, Part 2

So you’ve decided (or been told by management) that your VHD’s need to be able to work with both bare-metal and ESX (and Hyper-V). There are many different reasons why this is a valid thing to do. The major ones being portability between different hardware and the ability to use some kind of replication software (DFS-R for example) to copy your VHD to another site and automatically bring servers up there.

If you read Part 1 of this post you now know how to get your image to work between bare-metal and Hyper-V. If Hyper-V is necessary in your environment you can really do that piece before or after the process I’m about to explain. It’s kind of adjunct process.

The process for making your VHD/vDisk work with both bare-metal and ESX is mostly straightforward. It’s not the simplest procedure, but it’s the one I’ve found that consistently works.

The first thing you want to do is go to and download the V2V Converter. You can use XenCovnert to go P2V or V2P or use the VMWare Converter to do the same, but those utilities all insert their own little functionality which manages to break your VHD. the V2V converter does a block level conversion from an ESX .vmdk to a standard .vhd and vice-versa.

You’ll want to install this utility on the Citrix Provisioning Server, or wherever you’re hosting your vDisks.

  1. Boot your vDisk into Private Mode on one of your bare metal servers
  2. Copy the PVS Target Device software to a local temp directory. Also copy any other software/drivers you might need. When you convert the .vhd to a .vmdk you will NOT have any network access. (You could also mount your .vhd on the PVS server directly and copy the files into the temp location.)
  3. Shut down the bare-metal server and physically copy the .vhd file to a temp location on the PVS server.
  4. Run Starwinds V2V Image Converter.
  5. Browse to the source VHD and click Next
  6. Select VMWare Growable Image and click Next (assuming you thin-provision)
  7. IMPORTANT: Select IDE disk, even if it’s not.
  8. Select the name/location of the output .vmdk
  9. Let the conversion go…
  10. Install the vShphere client on your PVS server. Open it and log into vCenter.
  11. Browse to your datastores and create a Folder for your temp VM.
  12. Upload the .vmdk to this folder
  13. Create a new virtual machine. Give it 2 processors, however much memory you want, specify the E1000 NIC (super important for PVS), then UNCHECK the “Connect at Power on” box. You don’t want this bad boy on the network.
  14. When you get to the Virtual Disk screen select “use an existing” and browse to your uploaded .vmdk in the store.
  15. Power on the VM
  16. Log into the console with cached creds.
  17. Install the VMWare tools to get the appropriate drivers installed.
  18. Go to Control Panel and uninstall the Citrix Provisioning Services Target Device x64 software.
  19. Reboot.
  20. Log back in and reinstall the Target Device Software from your temp location. You remembered to copy it in, right?
  21. When it’s done, make sure all your NICs are checked and click OK. You should see at least one that is inactive.
  22. Reboot
  23. Power off the VM
  24. Repeat the above steps to download the .vmdk from the datastore back to your local temp location.
  25. Open the StarWind V2V Image Converter.
  26. Select the new .vmdk file (you’ll probably see temp.vmkd and temp-s001.vmkd, select the temp.vmdk)
  27. Select the destination as “MS Virtual PC growable image” and click Next. Again, this assumes you’re thin-provisioning.
  28. Select the output filename/location.
  29. When done, just copy the .vhd back into the PVS datastore and import it as usual.

As convoluted as that sounds, that’s really all you need to do. Your image will now be able to boot between both ESX and bare-metal. You’ll of course want to clean up the image a bit. Uninstall any hardware agents (HP SIM, etc.). And you’ll get some errors in the logs when you boot that it can’t find so-and-so hardware, but for the most part you should be set.

I’ve got my images booting between bare-metal (HP GL460c G6), Hyper-V, and ESXi 4. It’s pretty slick and took months to figure out the right order for everything but it now runs like a dream.

Of course if you followed the Citrix best-practices you were already using ESX or XenServer (same process, btw), so you probably don’t have this problem. But if you’re like most people don’t RTFM you’ll hit this pickle at some point.

Provisioning Server 5.6 and cross-platform VHD’s, Part 1

So if you’re like anyone, you actually buy new hardware. Today you’ve got a BL460c G5, tomorrow you can only buy G6’s. Or G7’s. You’ve also got an ESX environment and maybe even a Hyper-V server stashed somewhere. The beauty of Citrix PVS and streaming your VHD’s is that you can have bare metal just sitting there… sitting there doing nothing, and you can PXE boot it and be on your way.

You also like to stream your images to your VM’s, cause hey… You can virtualize your hardware, why not virtualize the OS you put on it?

And maybe you’ve got that Hyper-V server sitting over there, off to the side, and you want to stream your images to it as well, cause, you know, you can.

Soon enough you’ll be like me, and have 12 different images covering 3 different environments, and you realize what a total PITA is it to manage all these darn things. Because officially Citrix says they don’t support moving that image to different hardware and if you try it you get the dreaded BSOD.

But if you’re me, you don’t accept that answer…

The only way Citrix supports moving your image from one set of hardware to another is to use XenConvert and P2V it back to bare metal, then take the drives out and slap them in the other hardware. I’ve done it, and it’s not pretty, but it DOES work.

I found a relatively cool way online to make my G6 image work on Hyper-V. It was actually pretty straightforward:

  1. Do your normal build process for your image on the G6 and set it to Private mode.
  2. Copy the vhd from your PVS server over the network to your Hyper-V machine. Create a new VM with multiple processors, an appropriate amount of ram, and attach the VHD to it as it’s hard drive.
  3. Very important.. remove the default NIC and add the “Legacy NIC”
  4. Boot your VM, log into the machine and let it install all the stuff it needs to work.
  5. Uninstall the PVS Target Device Software, reboot, reinstall it, then when it comes up make sure to tell it to use all of the NICs, even the ones it says are unavailable.
  6. You see, the problem with PVS is that it loads all the nic’s it’s bound to when it starts the GUI boot. If it doesn’t load the actual device (not just the driver), then you get your BSOD.
  7. So after you reinstall the software, reboot again.
  8. Run XenAppPrep.exe /pvs from the command line and shut down your machine.
  9. Go back to the file system and copy the VHD back over to the PVS data store, add it thru the console and you should be good to go to boot the image from either your original physical or to Hyper-V.

Great! You’re done, right?

Well, not so much. Wouldn’t you rather have your image be hardware-independent? Wouldn’t you like to boot it thru ESX?

This process doesn’t remotely work for ESX.

Oh, and didn’t I mention? This only works with 2008. If you want 2003 or anything earlier, good luck. There aren’t any drivers for the legacy NIC in Window 2003.

If you want hardware independence, read on for Part 2.

SEP11 and Citrix PVS

So in my inherited Citrix Provisioning Server (PVS) 5.6 environment with SEP11, we recently noticed that none of the devices were showing up correctly in the SEP admin console (or at all), plus the Symantec Management Client service (which is set to automatic) wasn’t started. You could go into the services mmc (thru Server Manager or services.msc from the Run, which is my fave) and start it up manually and the device would appear in the console and everything was fine.

Reboot and you’d be back to where you started, which is typical with PVS disks in Standard Mode.

You see this in the Event Log:

Symantec Management Client service depends on a service in a group which starts later.


Detected circular dependencies auto-starting services. Check the service dependency tree.

Hey, that seems easy enough, right?

So after some googling and playing around with it I made a copy of my vhd, set it to Private mode and booted it up. Once up I went into the registry and browsed to


By default the value is set to NDIS. If you set it to NDIS2 everything starts working. So just run your Device Optimization Wizard, run your shutdown script (or xenappprep /pvs), shut down, then set your image back to Standard Mode and there you go.