Tip of the day: Ensure your Secure Reference Overrides Management pack gets backed up on a regular basis and after run as account modifications.
It doesn’t really matter how you back it up (manually export, powershell job, etc). Just make sure you have a copy of it, especially after importing and configuring other management packs.
Why is this important?
The Secure Reference Overrides MP contains all the configuration information that associates your Run As Accounts with your Run as Profiles. If you accidentally remove or delete this MP (it happens) you will see shit storm of alerts kick off for everything for data warehouse connection failures to discovery failures on your SQL instances. If you look in at your run as account configurations there wont be any accounts associated with profiles and when you try and add one you’ll see an error that looks like this.
First thing you should probably do in this scenario is send an email to the different technical teams giving them a heads up to monitor their own systems for a few min so they don’t start spamming you with inquires as to why your monitoring tool is spamming them. Then have a sip of coffee, locate your backup and re-import it from disk using the wizard. Once imported check to make sure your run as profiles have accounts associated with them. If you have made config changes or imported new MP’s since the backup you will have to re-config the run as accounts to accommodate those changes.
Once you have done that you should see the monitor based alerts close themselves automatically. Any remaining alerts will have to be investigated individually.
How to avoid this scenario?
The most common reason scenario occurs is when the Administrator removes a management pack that references the Secure Reference Override MP. The wizard will prompt you to remove the SRO MP without giving any warning that doing so will remove all run as profile configurations. It’s very painful, but basically you have to export the SRO MP, modify the dependencies to whatever MP you plan on deleting then re-import it as a new version. You can also accomplish this with powershell of course. This article gives a good overview of both techniques.
Overall its an easy concept and fairly easy to avoid but it seems like a lot of admins find themselves in this scenario just because they are not really aware of what this MP does and its importance. Best practice is to backup all your MP’s of course but keeping a recent copy of this one specifically may save you from an outage and a bunch of wasted time configuring all your run as accounts again.