Thursday, December 5, 2019

Cloud Management Gateway connection issues when using AzureAD authentication - StatusCode=401 StatusText=CMGConnector_Unauthorized

Hi All,

Just dropping a quick post regarding an issue we encountered recently when setting up a new Cloud Management Gateway and attempting to use only AzureAD based authentication.

In our case we were using Intune to deploy the Configuration Manager client, and the CCMSetup service was getting installed but the CCMSetup.log was displaying some of the following errors when trying to perform the installation:
RetrieveTokenFromStsServerImpl failed with error 0x87d0027e

Failed to get CCM access token and client doesn't have PKI issued cert to use SSL. Error 0x80070002

DownloadFileByWinHTTP failed with a non-recoverable failure, 0x87d00455

[CCMHTTP] ERROR INFO: StatusCode=401 StatusText=CMGConnector_Unauthorized

We were expecting to also find an CCM_STS.log file on the Managment Point, but it was not present at all. Digging in to the IIS traffic logs, we noticed that attempts to access the CCM_STS site were receiving a 302 response, indicating a redirect was occurring.

This specific server we were using to hold the MP role also had the Application Catalog role installed (this was slated to be removed in the very near future). The Application Catalog role configured an IIS redirect on the default web site so that all requests to the server were getting redirected to the Application catalog. Simply disabling the redirect and restarting IIS was enough to get our client install working across the CMG using AAD authentication with no PKI required. I'd wager that this issue is extremely unlikely to affect another ConfigMgr environment as most folks have likely removed all of their application catalog roles by now, but I figured this was such an odd scenario that it would be worth a blog post in case it can help someone else out.


Wednesday, February 13, 2019

High CPU utilization by CCMExec after upgrading to SCCM Build 1810

Hi All!

Encountered another issue today that I felt was worth a blog post since it happened after upgrading to build 1810 and there were very few Google hits on the error message.

We were alerted to an issue on a handful of workstation where the CCMExec and WMI services were consuming 100% of the CPU.

Taking a brief look at the client logs, I noticed that the M365AHandler.log was constantly rolling over nearly every single second.
The log was showing error messages like the following:
Running: "C:\Windows\system32\CompatTelRunner.exe" -m:appraiser.dll -f:DoScheduledTelemetryRun ent
Executing command line: Run Appraiser
CreateProcess failed. Code(0x80070002)
Command line execution failed (80070002)
CommandLine.Execute() failed.
CM365ASystemTask:RunAppraiser() failed. 0x80070002.

A quick search for C:\Windows\system32\CompatTelRunner.exe found that the file did not exist.

The solution for us was to install KB2952664. This update was previously deployed in our environment, but it appears these systems experiencing the issue were not targeted with it. Installing the update caused the CompatTelRunner.exe file to be created and the SCCM component was able to run it successfully.

Tuesday, June 12, 2018

0x800706d9 - There are no more endpoints available from the endpoint mapper

Just adding a quick post with a recent issue we ran in to on some of our endpoint devices. When trying to download SCCM client policy, we were seeing the error message "0x800706d9 - There are no more endpoints available from the endpoint mapper" in the datatransferservice.log.

Coincidentally, these devices had been recently upgraded to Windows 10.

The root cause for the issue turned out to be that the Windows Firewall service was disabled. At some point a technician must have decided that was a necessary change, but (at least in Windows 10) BITS downloads will fail unless the Windows Firewall service is running.


Friday, March 9, 2018

Resolving issues with user policy downloads failing due to large Kerberos token sizes

Hi All!
First post in a long time, but we just solved an issue in our production environment that others may run in to so I figured I would share the solution.

We were having issues with some users not receiving an application that was being deployed to a user collection. We could tell that the users were not downloading the policy object that would install the application, because we could see the following errors in the policyagent.log file when attempting to perform a user policy refresh:
 Synchronous policy assignment request with correlation guid {109B537A-194F-4171-A803-  5022A6C7D27F} for User $UserGUIDHere completed with status 80070005


We could correlate those errors with the following messages in the IIS log on the management point:
  CCM_POST /ccm_system_windowsauth/request - 80 - $IPAddressHere ccmhttp - 401 2 5 0


We checked in with our Microsoft PFE and he said it looked like we were seeing issues due to the large Kerberos tokens some of our users have due to large numbers of group memberships. We have configured a GPO in our environment that increases the max token size, but he pointed out a link to us that matched up with what we were seeing:



The full path of the registry keys is HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters. The keys have to be added as DWORD's. Their description says



MaxFieldLength Sets an upper limit for each header. See MaxRequestBytes. This limit translates to approximately 32k characters for a URL. Default Value – 16384, Range 64 - 65534 (64k - 2) bytes



MaxRequestBytes -Determines the upper limit for the total size of the Request line and the headers. Its default setting is 16KB. If this value is lower than MaxFieldLength, the MaxFieldLength value is adjusted.

Default Value – 16384, Range 256 - 16777216 (16MB) bytes

We configured the registry keys with the following values:
MaxFieldLength: 65534
MaxRequestBytes: 16777216

We also had to reboot the server before the changes would take effect, simply restarting IIS was not enough to see a change in the client behavior.

After reboot we again tried to do a user policy refresh from the client and were successful and no longer saw the 80070005 errors in the policyagent.log


Thursday, September 14, 2017

How to find all SCCM packages that have "Allow this package to be transferred via multicast" enabled using Powershell

This post was based on a question I came across online that I thought might be simple, but it wasn't as straightforward as I had hoped when looking at the Powershell output for Get-CMPackage.

What the person was trying to do is list all packages that were enabled for transfer via multicast. I took a test package from my lab environment and listed all of it's properties using Powershell:

Get-CMPackage -Id $ID | Select-Object -Property *

Although there is no obvious property for multicast transfer, I could see that every time I checked or unchecked the box for multicast transfer, the value for the PkgFlags property was changed.

I took a look at the MSDN documentation for the SMS_PackageBaseClass and found that while there are some values listed for PkgFlags, there was no value listed for handling multicast transfer:

I stumbled across an old post on MyITForum that explained that the multicast value (27) was undocumented.

I was able to take that information and combine it with a post Greg Ramsey had made on checking package properties with Powershell and put together this short snippet of code:

Get-CMPackage | ForEach-Object {
  if ($_.pkgflags -eq ($_.pkgflags -bor 0x8000000)) {
  "$($_.name)"
  }
}

If you run that bit of Powershell it will list the names of each package that is configured for Multicast.

Monday, July 3, 2017

Digging in to the new System Center Updates Publisher Preview

Lo and behold Microsoft has finally released a new version of the System Center Updates Publisher (AKA SCUP) that was last updated in 2011. It is now known simply as System Center Updates Publisher, and includes a release month indicator for each new version (currently June Preview). I don't see any info that indicates this is unsupported by Microsoft despite the Preview designation.

You can read the official blog post from Microsoft announcing the new version here.

The main focus for the update according to Microsoft was enabling the use of SCUP on Windows 10 and Server 2016, but I wanted to dig in to the tool a little bit and see what else was new under the hood, and I did in fact find that they fixed or at least improved some of the more annoying quirks that existed in the 2011 version.

First thing that I noticed was fixed was that when you launch the installer it now properly prompts for elevation in a UAC prompt, where the 2011 version simply threw an error that it needed administrator permissions in order to install and required you to launch the installation from an administrator command prompt. It also does a pre-requisite check for .NET 4.5.2 and will direct you to install it if it is missing.

The installation process itself is completely silent and I had to check the Start menu to see the new icon and program group.

One other change that I noticed is that the application is now 64-bit and installs in C:\Program Files\Microsoft\UpdatesPublisher by default, where the 2011 version was a 32-bit app.

Upon first launch, SCUP now performs an update check to see if there are any newer version, lending weight to the theory that they will be updating SCUP more frequently than they have previously done.The log file for SCUP is still located in the users temp folder (C:\Users\$username\AppData\Local\Temp\UpdatesPublisher.log) and here we can see the update check as it is happening:

Once you are in the main window for the application, you can see that most of the user interface remains largely the same with the exception that the main panes are now known as workspaces. Here is the old UI (top) and the new UI (bottom)



They did add some new sections to the options menu that are helpful. First up, here is the new advanced options page which has given you a button to change the database location, in the 2011 version this simply showed a read-only file path that could only be changed by modifying the config files directly:


Next, here are the new logging options, which let you configure a maximum log file size, as well as a slider with 6 (!) levels to set the amount of log detail you want. Why they went with a slider that has no context as to what you are getting when you choose one of the options I have no idea, but I'd wager that this will be improved in a future update. In my brief testing, this seemed to function more as an on/off switch where anything other than all the way to the right didn't seem to generate any log entries at all.



 Finally, there is an updates section to the options menu that allows you to opt out of checking for updates, or opting out of preview builds if you desire:


Another common issue that folks run in to when using SCUP in a large environment is that it is only able to be ran by one user at a time on a given machine, if a second user tries to launch the application it will simply fail to launch and give no indication that there was an error. I tested this scenario with the new release of SCUP and was pleasantly surprised to find that they have added an error dialog to give you an indication of what the problem is:

I also wanted to take a look at the format of the SCUP.exe.config file (now known as UpdatesPublisher.exe.config) to see if there were any changes made to it and was surprised to see that the structure of the file has changed significantly. You can see the embedded version of each file below with the old version on top and the new version on the bottom. The new config file (while still being written in XML) is significantly shorter and contains some new references to an "Entity Framework" that didn't exist before. It seems that most of the configuration data must have been moved in to the database itself.



While that is all of the changes in the new version of SCUP that I have noticed so far, I plan to spend some additional time investigating it in my lab environment over the next week or so, so check back for any updates to this post. As always, please reach out and connect with me using the Twitter or LinkedIn links on the right side of this page, I'd love to hear some feedback on my posts if they are helpful to you. Thanks! Matt

Thursday, June 29, 2017

Managing Configuration Manager BITS jobs with Powershell

I wanted to share some of the Powershell functions that I've created for managing the BITS jobs that are created when an SCCM client initiates a content download. I've added all of these to my Powershell profile so that I always have them loaded in my Powershell session. I should also mention that I don't consider myself a Powershell expert, so there are likely things being done in these functions which aren't considered best practice (using write-host for example) but they definitely get the job done.

All of these functions leverage PsExec since the BITS Powershell cmdlets don't support remote computer usage. They each expect to find a copy of PsExec.exe at the root of the C drive so make sure you have a copy placed there, or modify the functions to fit your use case. They all use the mandatory parameter -computername, so make sure you specify that when using the function.

First up is my function Get-BITSJobs, which simply gets a list of all of the SCCM BITS jobs on a remote computer and returns a list of them, including file count and size information:

Function Get-BITSJobs
{
Param(
   [Parameter(Mandatory=$true)]
   [string]$computername
)

& 'C:\PsExec.exe' \\$Computername -s bitsadmin.exe /list /allusers | Select-String -Pattern "CCMDTS Job"

}

Next is Set-BITSJobsForeground which will take all of the SCCM jobs on a remote computer and set them to foreground priority. This has often come in handy if a machine has been configure to throttle BITS download speeds, but for some reason I need that computer to finish it's downloads as fast as possible:

Function Set-BITSJobsForeground
{
Param(
   [Parameter(Mandatory=$true)]
   [string]$computername
)

[string]$jobs = & 'C:\PsExec.exe' \\$Computername -s bitsadmin.exe /list /allusers | Select-String -Pattern "CCMDTS Job"

If($jobs -ne "")
    {
    $arrjobs = $jobs.Split("`{*`}") | select-String -Pattern "-"
    Foreach ($job in $arrjobs)
        {
                 & 'C:\PsExec.exe' \\$Computername -s bitsadmin.exe /setpriority "`{$job`}" foreground        
        }
    }
Else {Write-Host "No Jobs"}
}

Next is Set-BITSJobsComplete, this will mark all SCCM jobs as completed, including any that are in error status. It then will restart the SCCM client if you use the parameter -RestartSCCMService after which the client will restart the downloads where they left off. I have generally used this for when a machine has it's downloads stuck in an error state and I need to get the jobs to start again:

Function Set-BITSJobsComplete
{
Param(
   [Parameter(Mandatory=$true)]
   [string]$computername,
   [switch]$RestartSCCMService
)

[string]$jobs = & 'C:\PsExec.exe' \\$Computername -s bitsadmin.exe /list /allusers | Select-String -Pattern "CCMDTS Job"

If($jobs -ne "")
    {
    $arrjobs = $jobs.Split("`{*`}") | select-String -Pattern "-"
    Foreach ($job in $arrjobs)
        {
         & 'C:\PsExec.exe' \\$Computername -s bitsadmin.exe /complete "`{$job`}"
                
        }

    If ($RestartSCCMService)
     {
      get-service -ComputerName $computername -Name CcmExec | Restart-Service
     }
    }
Else {Write-Host "No Jobs"}
}

Hopefully you will find these functions useful, I was using them often enough that it made it worth the effort to turn them in to functions. Try adding them to your Powershell profile so that they will always be accessible to you whenever you have an open Powershell window.