Configuration Manager policy and content downloads failing over the VPN AKA Error: 0x80200024, Description The job is not making progress.

Yet another bizarre troubleshooting exercise today that I think is worth sharing since due to the Covid-19 pandemic more and more users are working from home and using the VPN.

We were alerted to this by our application deployment teams who noticed higher than normal numbers of computers reporting an unknown status during deployments. We were able to make a correlation that the majority of these devices were users at home connected via VPN (PaloAlto GlobalProtect in our case).

Upon inspecting the datatransferservice.log on some of the client workstations, it was apparent that the policies for the deployment were not successfully being downloaded, with the BITS job reporting error 0x8020024. We also noticed this same error in ccmsetup.log for clients that were attempting to perform a client upgrade (we also installed ConfigMgr build 2002 the previous weekend):

I started reviewing the traffic logs using Wireshark to try and get a better idea of what was happening at the network level, and once I isolated the traffic it became pretty apparent where things were going wrong:

We can see the original GET request for the file, and then an immediate response from the server of HTTP 416 which corresponds with "requested range not satisfiable. Doing some Google searching turned up a few different forum threads (unrelated to ConfigMgr) where PaloAlto firewalls were blocking multithreaded downloads and sending the 416 response as if they were the server iteself. We were able to point our security team to a knowledge base article from PaloAlto with the necessary configuration changes.

Once they made the change to the firewall, our downloads started to complete almost immediately, and the unknown computer count on our deployments began to decrease rapidly.

WSUS Error: Exception in SimpleAuth.GetAuthorizationCookie: System.Data.SqlClient.SqlException (0x80131904): DB has reached allowed max number of local computers

So another fairly obscure issue occurred today that needs some better search results for the very few that would likely experience this issue.

We were starting to see WSUS scan failures reports on some of our client devices when reviewing the  built-in report Scan 1 - Last Scan States by Collection. In the report, the error displayed was:
Same as SOAPCLIENT_SOAPFAULT - SOAP client failed because there was a SOAP fault for reasons of WU_E_PT_SOAP_* error codes.

When we reviewed the SoftwareDistribution.log file on the SUP, we saw several instances of the error "w3wp.42 SimpleAuthImplementation.GetAuthorizationCookie EventId=400,Type=Error,Category=WsusService,Message=DB has reached allowed max number of local computers". We manage a large environment and are close to 100k clients in this ConfigMgr site which is the default number of devices that can be managed with a single WSUS server. Even though we have multiple SUP servers, they are all sharing a single database and all follow that same 100k total client limitation.

Another cause of this issue was that we had fallen behind on our SUP maintenance that typically removes computers that haven't synced within 30 days, leading to a larger than normal accumulation of machines.

We resolved the issue by immediately running our standard WSUS cleanup process (starting with the built in cleanup wizard, and finishing by declining superseded updates and defragmenting the database). We also have a plan to increase the number of devices that can be connected to WSUS to the maximum allowed when using Configuration Manager (150K) by executing the following Powershell code:

[reflection.assembly]::LoadWithPartialName("Microsoft.UpdateServices.Administration")

$config = (Get-WsusServer -Name WSUS.Server.Name -PortNumber 8530).GetConfiguration() $config.MaximumAllowedComputers = 150000

$config.Save()  

Hopefully this information will help anyone else that happens to run in to this limitation of WSUS, and most importantly, always run your WSUS maintenance :)

Cloud Management Gateway connection issues when using AzureAD authentication - StatusCode=401 StatusText=CMGConnector_Unauthorized

Hi All,

Just dropping a quick post regarding an issue we encountered recently when setting up a new Cloud Management Gateway and attempting to use only AzureAD based authentication.

In our case we were using Intune to deploy the Configuration Manager client, and the CCMSetup service was getting installed but the CCMSetup.log was displaying some of the following errors when trying to perform the installation:
RetrieveTokenFromStsServerImpl failed with error 0x87d0027e

Failed to get CCM access token and client doesn't have PKI issued cert to use SSL. Error 0x80070002

DownloadFileByWinHTTP failed with a non-recoverable failure, 0x87d00455

[CCMHTTP] ERROR INFO: StatusCode=401 StatusText=CMGConnector_Unauthorized

We were expecting to also find an CCM_STS.log file on the Managment Point, but it was not present at all. Digging in to the IIS traffic logs, we noticed that attempts to access the CCM_STS site were receiving a 302 response, indicating a redirect was occurring.

This specific server we were using to hold the MP role also had the Application Catalog role installed (this was slated to be removed in the very near future). The Application Catalog role configured an IIS redirect on the default web site so that all requests to the server were getting redirected to the Application catalog. Simply disabling the redirect and restarting IIS was enough to get our client install working across the CMG using AAD authentication with no PKI required. I'd wager that this issue is extremely unlikely to affect another ConfigMgr environment as most folks have likely removed all of their application catalog roles by now, but I figured this was such an odd scenario that it would be worth a blog post in case it can help someone else out.

High CPU utilization by CCMExec after upgrading to SCCM Build 1810

Hi All!

Encountered another issue today that I felt was worth a blog post since it happened after upgrading to build 1810 and there were very few Google hits on the error message.

We were alerted to an issue on a handful of workstation where the CCMExec and WMI services were consuming 100% of the CPU.

Taking a brief look at the client logs, I noticed that the M365AHandler.log was constantly rolling over nearly every single second.

The log was showing error messages like the following:

Running: "C:\Windows\system32\CompatTelRunner.exe" -m:appraiser.dll -f:DoScheduledTelemetryRun ent

Executing command line: Run Appraiser

CreateProcess failed. Code(0x80070002)

Command line execution failed (80070002)

CommandLine.Execute() failed.

CM365ASystemTask:RunAppraiser() failed. 0x80070002.

A quick search for C:\Windows\system32\CompatTelRunner.exe found that the file did not exist.

The solution for us was to install KB2952664. This update was previously deployed in our environment, but it appears these systems experiencing the issue were not targeted with it. Installing the update caused the CompatTelRunner.exe file to be created and the SCCM component was able to run it successfully.

0x800706d9 - There are no more endpoints available from the endpoint mapper

Just adding a quick post with a recent issue we ran in to on some of our endpoint devices. When trying to download SCCM client policy, we were seeing the error message "0x800706d9 - There are no more endpoints available from the endpoint mapper" in the datatransferservice.log.

Coincidentally, these devices had been recently upgraded to Windows 10.

The root cause for the issue turned out to be that the Windows Firewall service was disabled. At some point a technician must have decided that was a necessary change, but (at least in Windows 10) BITS downloads will fail unless the Windows Firewall service is running.

Resolving issues with user policy downloads failing due to large Kerberos token sizes

Hi All!
First post in a long time, but we just solved an issue in our production environment that others may run in to so I figured I would share the solution.

We were having issues with some users not receiving an application that was being deployed to a user collection. We could tell that the users were not downloading the policy object that would install the application, because we could see the following errors in the policyagent.log file when attempting to perform a user policy refresh:

 Synchronous policy assignment request with correlation guid {109B537A-194F-4171-A803-  5022A6C7D27F} for User $UserGUIDHere completed with status 80070005

We could correlate those errors with the following messages in the IIS log on the management point:

  CCM_POST /ccm_system_windowsauth/request - 80 - $IPAddressHere ccmhttp - 401 2 5 0

We checked in with our Microsoft PFE and he said it looked like we were seeing issues due to the large Kerberos tokens some of our users have due to large numbers of group memberships. We have configured a GPO in our environment that increases the max token size, but he pointed out a link to us that matched up with what we were seeing:

https://blogs.msdn.microsoft.com/ashishsingh/2010/04/03/windows-authentication-for-accounts-with-large-kerberos-tickets-may-not-work-despite-having-maxtokensize-in-place/

The full path of the registry keys is HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters. The keys have to be added as DWORD's. Their description says

MaxFieldLength - Sets an upper limit for each header. See MaxRequestBytes. This limit translates to approximately 32k characters for a URL. Default Value – 16384, Range 64 - 65534 (64k - 2) bytes

MaxRequestBytes -Determines the upper limit for the total size of the Request line and the headers. Its default setting is 16KB. If this value is lower than MaxFieldLength, the MaxFieldLength value is adjusted.

Default Value – 16384, Range 256 - 16777216 (16MB) bytes

We configured the registry keys with the following values:
MaxFieldLength: 65534
MaxRequestBytes: 16777216

We also had to reboot the server before the changes would take effect, simply restarting IIS was not enough to see a change in the client behavior.

After reboot we again tried to do a user policy refresh from the client and were successful and no longer saw the 80070005 errors in the policyagent.log

How to find all SCCM packages that have "Allow this package to be transferred via multicast" enabled using Powershell

This post was based on a question I came across online that I thought might be simple, but it wasn't as straightforward as I had hoped when looking at the Powershell output for Get-CMPackage.

What the person was trying to do is list all packages that were enabled for transfer via multicast. I took a test package from my lab environment and listed all of it's properties using Powershell:

Get-CMPackage -Id $ID | Select-Object -Property *

Although there is no obvious property for multicast transfer, I could see that every time I checked or unchecked the box for multicast transfer, the value for the PkgFlags property was changed.

I took a look at the MSDN documentation for the SMS_PackageBaseClass and found that while there are some values listed for PkgFlags, there was no value listed for handling multicast transfer:

I stumbled across an old post on MyITForum that explained that the multicast value (27) was undocumented.

I was able to take that information and combine it with a post Greg Ramsey had made on checking package properties with Powershell and put together this short snippet of code:

Get-CMPackage | ForEach-Object {
  if ($_.pkgflags -eq ($_.pkgflags -bor 0x8000000)) {
  "$($_.name)"
  }
}

If you run that bit of Powershell it will list the names of each package that is configured for Multicast.