So I woke up one day last week to a critical ticket from the Helpdesk. We were getting “tons of calls” from users of one of our Citrix applications that they couldn’t launch it. They’d log in through the WI and when they clicked to launch the app were getting the error: “The Group Policy Client Services failed to Log On. Access is Denied.”
The tickets were coming in like a flood but when I went to go check the application in the Citrix AMC I could see we already had about 200 users connected (normally it’s like 250). So, okay, the application itself and the Citrix servers were okay.
Now let me backtrack for a second and give a quick run down of our environment. We’ve got well over 50K users spread across about 35 Windows AD domains, all in one forest. Very few users have access to all of the domains and until the day after this incident happened I wasn’t one of them. But like any user I’ve got read access to all objects so after about a half hour or so of troubleshooting on the server side and not finding anything I went back to the users.
All of the users were in 2 different domains, both of which happened to be in the same physical location and their domain admins were in the process of migrating users from 1 domain into the other. So, hey, commonality!
My Citrix servers are using UPM 3.2.2 for Profile Management and we save the profiles to a file server at the main site and use DFS to replicate them to our secondary site for DR. Not much is allowed to be saved into their profiles, but there are a few tiny things that are needed for the application.
A google search turned up a few things to look at on the user accounts so I started looking at their Terminal Profile tab. Now if you know UPM at all you know it doesn’t play well with any other kind of profile management. We’ve got some GPO settings in place to try and disable the use of terminal services profiles, but there’s no real way to fully disable the use of them. We’ve had conflicts before, so it’s a process.
In this case all the users had the exact same settings on their Terminal Profile tab. It looked weird and after a few seconds of looking at them it became obvious. It was pointing to serversharedirectory. And that was it. You’d think it should have been pointing to something like servershare%username% or some such. So this meant those users had Mandatory user profile configured. Do you know the one thing that UPM doesn’t work at all with?
Yep, Mandatory user profiles.
After that, the fix was easy. We had to remove the settings from their account, replicate, delete their UPM profile on the share, then they were fine.
Local support said that those were “old” settings and no longer used, so we could delete them without any problem. We still haven’t answered the obvious question: If we changed nothing, and local support changed nothing on their domain, then why did this setting suddenly matter or make a difference?
The best we can figure is some policy was enabled or disabled on either our domain or theirs that enabled that setting to be used, but I’ll be darned if we can figure out what.