Per direction of our IA department we recently updated the self-signed certificate on our SolarWinds Application Server (Windows Server 2008 R2 Enterprise platform) from 1024 bit to 2048 bit. Here are the instructions we followed: https://thwack.solarwinds.com/community/solarwinds-community/geek-speak_tht/blog/2012/10/23/getting-certificates-up-to-speed--updating-rsa-key-security
We carefully followed the process, but now our Network Configuration Manager is broken:
1. It will not download or upload Cisco startup or running configurations. The error message we receive is: "Start Transfer Error. See NcmBusinessLayerPlugin log for details" "Fix connection in Device Template." When we click on the "Fix connection" link in the Configuration Manager/Transfer Status tab, then navigate to the General Device Access tab, and we verify the settings and press "Test," we receive the error: "Unable to connect to polling engine (server name) on the relevant server. Verify that NCM 7.4 or later is installed on the server." We are running NCM 7.4.
2. We cannot edit nodes, even if we delete them and re-add them. When we attempt to edit a node, we receive the following message: "There was an error retrieving data from SolarWinds Information Service" and "Invoke failed, check fault information."
3. We have several WMI (Windows) credentials stored. They can be verified in Settings/Windows Credentials/Manage Windows Credentials. However, when we add a new Windows Server 2008 R2 Enterprise node and select the WMI option from the Windows Servers: WMI and ICMP/Choose credential/<New Credential> arrow, no options are available. If we attempt to type the stored credential name manually, it still does not appear.
4. We CAN run default SolarWinds reports from the CONFIGS tab/Reports menu option in the web interface. However, none of our scheduled jobs will run. Everything (including the items listed above) was working fine until the certificate update.
These behaviors persist regardless of browser (Firefox 41.0.2 or Internet Explorer 10.0.32)
We opened a support case with SolarWinds three weeks ago and have been communicating with them daily via telephone and email. They have walked us through an array of troubleshooting efforts including registry fixes, but so far nothing is working.
The main log files we have been dealing with in our troubleshooting efforts are:
- BusinessLayerHost.log
- Core.BusinessLayer.log
- InformationService.log
- NcmBusinessLayerPlugin.log
- Orion.InformationService.log
- OrionPermissionChecker.log
- OrionWeb.log
The licensed products we are running are:
- Orion Platform 2015.1.2
- NCM 7.4 (NCM-NPM Integration 7.4)
- SAM 6.2.2
- NPM 11.5.2
- IPAM 4.3
- NTA 4.1.1
Also:
- DPA 10.0.0
- PM 2.1
And:
- SolarWinds Collector v2.12.38
- SolarWinds Job Engine v2.10.0
- SolarWinds Integrated Virtual Infrastructure Monitor v2.1.0
- SolarWinds Information Service v2015.1.6134
The Hot Fixes are also installed.
We are running Windows.NET Framework 4.5 and WinPcap 4.1.3.
Our SolarWinds Application Server and related servers (SQL and NTA) are located on a closed network with no direct access to the Internet.
The reason I've listed so many items is because the SolarWinds technical support and development teams have had us digging through all of them, weeding through logs, reports, program files and registry keys, and also repairing and uninstalling/re-installing nearly everything but to no avail. We are close to making a decision to completely tear down the server and rebuild it from bare-bones scratch as if it were new hardware fresh out of the box. We would really love to avoid that level of effort.
And no, we don't have the luxury of a test environment for this sort of thing, else we could have avoided this altogether in our production environment.
Has anyone else encountered this issue? If so, what was your solution?
Many thanks for your patience in reading and considering this problem.