Saturday, July 8, 2017

MP has discarded a report when processing Relay

You may see below error message for SMS_MP_CONTROL_MANAGER  in component status node under monitoring after upgrading SCCM Current Branch to 1702

MP has discarded a report when processing Relay.
Possible cause: Corruption or invalid user definition.
Solution: Check the logs to identify the cause. If the problem is persistent raise the issue with Microsoft support

Background history:
SMS_MP_CONTROL_MANAGER reporting MP has discarded a report when processing Relay error after upgrading SCCM CB to 1702.
All the MP's are reporting similar error since the SCCM branch upgrade.

<InstallDIr>\SMS\MP\OUTBOXES\\bad folder contains thousands of badrelay XML files.

Reviewing MP_Retry.log file (Can be found  C:\Windows\ccm\logs on the server where MP is installed) shows;
instance of MpEvent_DiscardingInventoryReport
Mp Task: saved bad report as D:\SMS\mp\outboxes\\bad\badRelayZGERVQ7W.XML

The reason for this error is, this is the On Demand request as per MP_Location which causes the above error. These are generated for the Packages where On Demand distribution is enable.

To find out which packages configured with On Demand distribution is enabled, run the below query in SQL Management Studio, then look for  PKG_DISTRIBUTE_ON_DEMAND value is 1.

   (PkgFlags&0x40000000)/0x40000000 AS PKG_DISTRIBUTE_ON_DEMAND 
  dbo.v_Package pkg

Once you have identified and removed the on demand distribution, the error will stop and you can see the number of logs in bad folder gradually go away. It will take few days before the folder gets empty.

Friday, July 7, 2017

SCCM report for packages where on demand distribution is enabled

This query will provide package name and on demand distribution status.
If on demand distribution status is set to 1, then on demand is enabled for that package.
SELECT  Name, 
   (PkgFlags&0x40000000)/0x40000000 AS PKG_DISTRIBUTE_ON_DEMAND 
FROM  dbo.v_Package pkg 
More SCCM custom reports can be found here

Saturday, June 24, 2017

Remote configuration failed on WSUS

On one of the recent patch Tuesday, our Software Update Point (SUP) had issues and failed to sync with below error messages;
Remote configuration failed on WSUS
wsyncmgr.log  shows;
Sync failed: WSUS server not configured. Please refer to WCM.log for configuration error details.. Source: CWSyncMgr::DoSync wsyncmgr.log

WCM.log shows;
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host~~   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)~~   --- End of inner exception stack trace ---~~   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)~~   at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)~~   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)~~   --- End of inner exception stack trace ---~~   at Microsoft.UpdateServices.Administration.AdminProxy.CreateUpdateServer(Object[] args)~~   at Microsoft.SystemsManagementServer.WSUS.WSUSServer.ConnectToWSUSServer(String ServerName, Boolean UseSSL, Int32 PortNumber)

The same SUP has been working fine until today. No configuration changes been made.
When I checked the software update point the CPU usage was so high that I cannot even launch the WSUS console.
It could be a coincident that the high CPU usage is because of the clients scanning for new updates.
Due to the high CPU usage, most of the clients were failed to complete the scan and as well as the upstream server failed to complete the update sync.

So I have noticed three main issues;
1. Upstream sync failed
2. Client failing to scan for the updates
3. Unable to launch the WSUS console

So to get the upstream sync successful, it is required to bring the CPU utilisation down by blocking the client devices to scan the updates.

Note: WSUS sync will stop working because of so many reasons. However, if it was working in the past and stopped working suddenly without any configuration changes to the infrastructure then most likely it would be an issue with the load on the server and server resources availability. First to eliminate the server load, block the clients to reach out to the SUP. Then test launching the WSUS console then try upstream server sync.Without blocking the client access to the SUP, even if we re-install WSUS and SUP we will have the same issue again.

To block client machines to scan for the updates, On Software Update Point;
go to C:\program Files\update services\webservices then re-name ClientWebService folder to something else.
This will make the Software Update Point unavailable to the clients. Once the sync is completed, then re-name the folder back to ClientWebService then restart the IIS services.
Now the upstream sync successfully completed and the clients also completed the scan successfully.

On the side note, make sure all the expired and superseded updates are removed from the WSUS database periodically. This is an important process in keeping the WSUS performance up. The more expired and superseded updates in the database, the slower the performance will be.

Saturday, April 29, 2017

Error: VSS_E_WRITERERROR_TIMEOUT. Error Code = 0x80,042,3f2

SCCM site database maintenance task failed with one of the below error;

After GatherWriterMetadata SMS Writer status = FAILED_AT_PREPARE_BACKUP.
Error: VSS_E_WRITERERROR_TIMEOUT. Error Code = 0x80,042,3f2.

ERROR: SQL Backup task failed. Error message - Asynchronous QueryStatus failed or did not finish properly.
STATMSG: ID=5052 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_SITE_BACKUP" SYS=sccb.w2k8.lab SITE=SCB PID=35284 TID=34312 GMTDATE=Wed Apr 26 23:25:37.402 2017 ISTR0="\\sccb.w2k8.lab" ISTR1="CM_CP1;" ISTR2="Asynchronous QueryStatus failed or did not finish properly." ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0
Error: Sql Server could not prepare for the Backup.

Current state of SQL backup: [failed]. Aborting it.
SMS_SITE_BACKUP 4/26/2017 11:25:37 PM 34312 (0x8608)

SMS_SITE_BACKUP failed. Please see previous errors.

Above errors can be seen in smsbkup.log under <CM installDir>\Program Files\Microsoft Configuration Manager\Logs

Below event viewer msgs can be seen on the site database server;

Event ID: 12346
Volume Shadow Copy Error: An error 0x8007000e, Not enough storage is available to complete this operation.
 was encountered while trying to initialize the Registry Writer.  This may cause future shadow-copy creations to fail.

Event ID: 12347
Volume Shadow Copy Service error: An internal inconsistency was detected in trying to contact shadow copy service writers.  The Registry Writer failed to respond to a query from VSS. Check to see that the Event Service and Volume Shadow Copy Service are operating properly, and please check the Application event log for any other events.

   Gathering Writer Data
   Executing Asynchronous Operation

   Execution Context: Requestor
   Current State: GatherWriterMetadata

If we investigate, further;
- COM+ Event System service is running
- Volume Shadow Copy service is running
- SMS_SITE_VSS_WRITER is running and can restart the service without any error
- On the command line run vssadmin list writers  and should list all the writers

When I ran vssadmin list writers on the site server, I have received various vss writers names and with last error. All of the writers showed stable except SMS Writer.
For the SMS Writer, state: [7] Last error: Timed out.

Then ran vssadmin list writers on the Site database server and received empty list.

A quick google showed me various articles to re-register the dll’s to initialise the writers.
However decided to restart the server as it has been up and running for good few months.
After the restart, I re-ran the vssadmin list writers  and this time it showed various writers.

Now initiated backup by starting the SMS_SITE_BACKUP service.
Even though I have received "After GatherWriterMetadata SMS Writer status = FAILED_AT_PREPARE_BACKUP. Error: VSS_E_WRITERERROR_TIMEOUT. Error Code = 0x80,042,3f2." Error in the beginning of the backup process, it has completed the site backup successfully.

The reason for the site database backup failure was, on the site database server the Registry Writer was not present. For some reason all the VSS writers had disappeared from the database server.
After restarting the server, the writers were back and everything worked as normal.

Tuesday, April 11, 2017

How does SCCM Current Branch automatic client upgrade work

Once new client is promoted and deployed to the production machines based on the client upgrade settings, the client machines will upgrade to new client.
If we want to monitor the process as it happens, follow the below steps to see and monitor the progress. Some times it will help us to troubleshoot upgrade issues.

1. After deploying the client to the production using hierarchy settings, on the client machine run Machine policy Retrieval & Evaluation Cycle
2. The existing agent will detect the client version and start processing the upgrade request. Open ccmsetup.log and we see these initial entries;

3. Existing client will query for ccmsetup.exe and download ccmsetup.exe from assigned MP;

4. The machine policy will check for existence of Configuration manager client upgrade task, if doesn’t exist then Configuration manager client upgrade task will be created in scheduled task;

5. Now open the task scheduler and we can see Configuration manager client upgrade task. The newly created scheduled task will run on random timings.

6. For testing or immediate client install, Run the Configuration manager client upgrade task from task scheduler
When we ran the task scheduler, ccmsetup.log (C:\Windows\ccmsetup\Logs)will show ccmsetup is launched by windows task scheduler for client upgrade;

7. Once the setup is initiated, the scheduled task will be deleted from the Scheduled tasks
8. Wait for 5-10 min to complete the installation and check the ccmsetup.log file. 
    It should have ccmSetup is existing with return code 0