Yet another bizarre troubleshooting exercise today that I think is worth sharing since due to the Covid-19 pandemic more and more users are working from home and using the VPN.
We were alerted to this by our application deployment teams who noticed higher than normal numbers of computers reporting an unknown status during deployments. We were able to make a correlation that the majority of these devices were users at home connected via VPN (PaloAlto GlobalProtect in our case).
Upon inspecting the datatransferservice.log on some of the client workstations, it was apparent that the policies for the deployment were not successfully being downloaded, with the BITS job reporting error 0x8020024. We also noticed this same error in ccmsetup.log for clients that were attempting to perform a client upgrade (we also installed ConfigMgr build 2002 the previous weekend):
I started reviewing the traffic logs using Wireshark to try and get a better idea of what was happening at the network level, and once I isolated the traffic it became pretty apparent where things were going wrong:
We can see the original GET request for the file, and then an immediate response from the server of HTTP 416 which corresponds with "requested range not satisfiable. Doing some Google searching turned up a few different forum threads (unrelated to ConfigMgr) where PaloAlto firewalls were blocking multithreaded downloads and sending the 416 response as if they were the server iteself. We were able to point our security team to a knowledge base article from PaloAlto with the necessary configuration changes.
Once they made the change to the firewall, our downloads started to complete almost immediately, and the unknown computer count on our deployments began to decrease rapidly.