How To: Get Exchange 2013 back up after failed CU install

Exchange 2013 is a whole new beast when it comes to a stand alone Exchange server. A failed Exchange 2013 CU (Cumulative Update) can leave your Exchange server unresponsive to outside requests even though all Services are up and Powershell diagnostics are reporting everything working correctly. It leaves many Exchange Admins new to 2013 scratching their heads and attempting to restore from backup only to learn that a restore from backup does not work! What is this sorcery?! Stay calm. This is an easy fix and a lesson for new 2013/2016 Exchange admins that are used to the previous versions.

Brave New World

These are dark days for Exchange admins that work for small businesses with less than 75 users or so. Exchange is becoming more of an enterprise product and much less of a small business email platform. Microsoft wants you to go to Office 365 if you’re small and leave Exchange to the “big boys”. This is made evident by the level of internal complexity Exchange has accrued in the last two releases (2013, 2016). A new concept that was introduced with Exchange 2013 is Server Component States (Read this to get more details). This new function is largely a Datacenter enhancement for managing unhealthy members of an Exchange cluster to prevent bottlenecks in performance as a result of a single unhealthy server grabbing client and database requests.

Cut to the end, will ya? How do I kill it?

So with the explanation of Server Component States behind us, we come to the one time this becomes a reality for an Exchange Admin with one Exchange server. A failed CU update. During a failed CU update the “Requesters” (Typically Maintenance and Functional) shutdown all Server Component States. What happens when this is the case? Well, the Exchange server appears to be functioning exactly like it should. All Services are up, all databases are mounted, and all web resources respond normally. However, mail flow is dead, no Outlook clients can connect or they do sporadically, and all Activesync functions are stalled. So how do you check for this state? One simple command:

Get-ServerComponentState –Identity <Server>

This will list all of the component states for that server. After a failed CU update it will show many “inactive”, if not all of them.

So how do I get it back? With a couple commands depending on the “Requesters” involved. (This example is with Functional and Maintenance)

Set-ServerComponentState -Identity <Server> -Component ServerWideOffline -State Active -Requester Functional

Set-ServerComponentState -Identity <Server> -Component ServerWideOffline -State Active -Requester Maintenance

How do you check which requester’s caused the inactive states? Well that is a bit tougher to find out. Two commands, one to set a variable to the Component ServerWideOffline and the second to pipe that into a call to Active Directory to get the value of the Requesters.

<variable> = Get-ServerComponentState –Identity <Server> -Component ServerWideOffline

<variable>.RemoteStates

You can also find the details in the Registry of the Exchange server or in Active Directory.

Registry Location =

\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\ServerComponentStates

Active Directory (ADSI Edit) Location =

Configuration > Services > Microsoft Exchange > Organization Name > Administrative Groups > Exchange Administrative Group (FYDIBOHF23SPDLT) > Servers > (Server Name).

Right click the Server Name and select Properties.  Scroll down to “msExchComponentStates”. Double click the entry and it will give you all the values added to this attribute. You want to look for the entries 1:ServerWideOffline:(Requester):(NumberStrings). Those are the Requesters you will need to re-enable the Exchange Server Component States.

ADSI_Edit_msExchComponentStates

This is why a restore from backup does not work. The component states are in Active Directory and the Exchange server will use the newest Component state entry between the Registry and Active Directory. So if you restore from backup, the registry entry will get overwritten with the entry from Active Directory because the Registry Entry from the backed up Exchange server will have an older Component State entry than the Active Directory one.

I hope this helps some Exchange Admin that is freaking out about their Exchange Server after a failed CU update. I was there once. It is not a fun place to be.

Cheers.

 

 

12 thoughts on “How To: Get Exchange 2013 back up after failed CU install

  1. After a failed CU 16 this has resolved the issue of no mailflow. Thanks so much for posting I can now troubleshoot why it failed in peace knowing that things are still working. All the best.

    Like

  2. Any idea if we could use the /m:RecoverServer switch to bring an Exchange 2013 server back to its previous CU? For example, if we’re upgrading from CU14 to CU15 and needed to revert back to CU14, can I use the CU14 install files with /m:RecoverServer to bring it back to CU14? Or am I stuck having to uninstall Exchange entirely and reinstall with CU14? We are generally required to have a ‘fallback’ plan, and since CU14 was the original version of 2013 we installed, I’m not entirely sure just how complicated that might be. We do use DAGs, if that makes any difference.

    I’ve been tempted to try /m:RecoverServer to roll a lab server back to CU14, but haven’t had the nerve to try it just yet. I know /m:Upgrade won’t work, because setup.exe is smart enough to know that CU15 is newer than CU14 (yes, I tried that out of curiosity).

    Liked by 1 person

    • Unfortunately I haven never had a need to use the switch. In a DAG situation I would take one of the Servers out of the DAG, then restore from backup to an earlier CU. Then move the service to the newly restored DAG member and remove the others. Upgrading the others and moving them all back into the DAG once the roll back was complete.

      Like

  3. Just had this problem last night. Unfortunately my search strings did not lead me to this article until *after* I had a call in to MS Tech support. $500 later they identified server component state as being inactive. Armed with the name of the part that was needing fixed I was able to find this page. Too late to fix my issue but at least I have learned what went wrong! Thanks!

    Like

  4. I try and install the CU and I get this. I think I have powershell in the right mode. Any suggestions or help? Thanks
    Microsoft.PowerShell.Commands.ServiceCommandException: Service ‘Application Identity (AppIDSvc)’ cannot be configured due to the following error: Access is denied

    Like

Leave a comment