A customer with 10,000+ non-persistent VDI users got in touch about and long running issue that was generating a lot of noise on the service desk, i.e. approx. 50-100 calls per day, and affecting the productivity of around 1% of their workforce.
The issue was described as "Faulty" VM's. In essence what was happening was that Horizon was assigning the user a machine and once they were logged on, the environment would be incomplete, i.e. missing shortcuts, folder redirections etc.
On first impressions it looked like it was just a case that DEM wasn't working (i.e. not applying user profile archives or even policy) but that was quickly ruled out because it was observed that the DEM agent service was running but it was seen that the registry key for the DEM profile archive share was not present in multiple of the sample cases reported. As such DEM was merely unable to find the users DEM configuration path or DEM profile archive path. There were also a number of group policy applied settings missing.
Below are the reg keys that were missing that showed the GPO pushing DEM settings were not applying.
(These were user config registry settings applied via a GPO linked to the OU housing the VDI computer objects).
The Horizon Agent VDM event logs did not produce any clues, however the windows event logs showed a re-occuring error 1058 on the problem machines; The error 1058 appeared to happen around boot time of the non-persistent VDI, as opposed to when the user logged on.
Users were typically able to work once the service desk reset their session and they logged onto another 'healthy' machine.
The processing of Group Policy failed. Windows attempted to read the file \\<domainname.local>\SysVol\<domainname.local>\Policies\<{ABC12345678-1234-ABCD-1234-GUID-ETC}>\gpt.ini from a domain controller and was not successful. Group Policy settings may not be applied until this event is resolved. This issue may be transient and could be caused by one or more of the following:
a) Name Resolution/Network Connectivity to the current domain controller.
b) File Replication Service Latency (a file created on another domain controller has not replicated to the current domain controller).
c) The Distributed File System (DFS) client has been disabled.
User Policy could not be updated successfully.
Digging into the above error deeper we confirmed DNS was always working fine, the VDI machine could resolve the DC, and vice versa without issue. AD sites and services had a subnet defined for the VDI subnet and a local domain controller was specified there.
A repadmin command on the domain controller showed the DC's were sync'ing fine with each other.
At no point were we unable to access the UNC path specified in the above errors. i.e. they were present and no permissions issues.
\\<domainname.local>\SysVol\<domainname.local>\Policies\<{ABC12345678-1234-ABCD-1234-GUID-ETC}>\
Other checks included making sure that the "Always wait for computer at startup and logon" group policy setting was applied (which it was) https://admx.help/?Category=Windows_10_2016&Policy=Microsoft.Policies.WindowsLogon::SyncForegroundPolicy
Plus checking the loopback policy was set correctly for VDI, which it was.
GPUpdate /force wouldn't resolve the issue, the VDI usually had to be reset and re-provisioned.
The root cause:
The error 1058 was down to "UNC hardening" a security feature added to Windows 10 releases a few years back.
By default Windows will now harden all UNC paths and this was breaking group policy processing at boot time.
Testing showed that if we ran the PoSh command below to set this registry and disable UNC hardening, we were then able to successfully complete a GPUpdate /force on affected machines.
Set-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\NetworkProvider\HardenedPaths" -Name \\*\SYSVOL -Value "RequireMutualAuthentication=0, RequireIntegrity=0" -Force
This makes senses because UNC hardening can trip access to DFSR UNC paths, and DSFR is what SYSVOL uses to host group policies and replicate them between domain controllers.
The solution:
The final solution was two pronged
1. To permanently add the above mentioned HardenPaths to the base image. This meant that whenever the machine boots and group policy processing is attempted, then the SYSVOL path would be 'weakened' and group policy processing would always be successful.
2. This is essential for security reasons - create a GPO that undo's the 'weaked' hardened path that we have specified in the base image. i.e. a registry preference or a script that remove's this registry key.
Add comment
Comments