I have been unable to manually kick off NCM jobs for the past couple days and it seems the log snippet below is showing in Core.BusinessLayer.log every time I try to run a job manually, not to mention an error showing up in the Web UI(Error states: ProvideFault failed, check fault information). The interesting part of the issue is none of the servers(1 web, 2 pollers, 1 db) have low disk space, especially the database which this error seems to be pointing toward. I found this thread referencing .Net and the CLR setting for the database but I don't know if this is the same symptoms.
Can anyone offer any assistance into what may be causing this to occur? I really appreciate the help in advance, thanks PJ Martin
2015-07-21 15:11:05,236 [76] ERROR SolarWinds.Orion.Core.Common.MessageUtilities - Exception caught in method ProvideFault
System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: Not enough storage is available to complete this operation. (Fault Detail is equal to An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is:
System.Data.SqlServerCe.SqlCeException: Not enough storage is available to complete this operation.
at System.Data.SqlServerCe.SqlCeCommand.ProcessResults(Int32 hr)
at System.Data.SqlServerCe.SqlCeCommand.InitializeCommand(CommandBehavior behavior, String method)
at System.Data.SqlServerCe.SqlCeCommand.ExecuteCommand(CommandBehavior behavior, String method, ResultSetOptions options)
at System.Data.SqlServerCe.SqlCeCommand.ExecuteReader(CommandBehavior behavior)
at SolarWinds.Collector.SqlCeHelper.ExecuteReader(SqlCeCommand command)
at SolarWinds.Collector.Repository.ConfigRepository.<EnumeratePollingDescription>d__16.MoveNext()
at SolarWinds.Collector.PollingController.PollingController.ResolveConditionRecursive(JobExecutionCondition condition, PollInformation pollInformation, UpdatePollingContextBehavior behavior)
at SolarWinds.Collector.PollingController.PollingController.JobNow(JobEx...).
2015-07-21 15:11:05,236 [76] ERROR SolarWinds.Orion.Core.Common.CoreErrorHandler - Error occured on Business Layer call. Type: System.ServiceModel.FaultException`1[[System.ServiceModel.ExceptionDetail, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]
Detail:
Type: System.ServiceModel.ExceptionDetail
HelpLink: ''
InnerException: ''
Message: 'Not enough storage is available to complete this operation.'
StackTrace: ' at System.Data.SqlServerCe.SqlCeCommand.ProcessResults(Int32 hr)
at System.Data.SqlServerCe.SqlCeCommand.InitializeCommand(CommandBehavior behavior, String method)
at System.Data.SqlServerCe.SqlCeCommand.ExecuteCommand(CommandBehavior behavior, String method, ResultSetOptions options)
at System.Data.SqlServerCe.SqlCeCommand.ExecuteReader(CommandBehavior behavior)
at SolarWinds.Collector.SqlCeHelper.ExecuteReader(SqlCeCommand command)
at SolarWinds.Collector.Repository.ConfigRepository.<EnumeratePollingDescription>d__16.MoveNext()
at SolarWinds.Collector.PollingController.PollingController.ResolveConditionRecursive(JobExecutionCondition condition, PollInformation pollInformation, UpdatePollingContextBehavior behavior)
at SolarWinds.Collector.PollingController.PollingController.JobNow(JobExecutionCondition condition, PollInformation pollInformation)
at SolarWinds.Collector.PollingController.PollingController.PollNow(String entityIdentifier)
at SolarWinds.Collector.PollingController.PollingControllerService.PollNow(String entityIdentifier)
at SyncInvokePollNow(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)'
Type: 'System.Data.SqlServerCe.SqlCeException'Action: 'http://schemas.microsoft.com/net/2005/12/windowscommunicationfoundation/dispatcher/fault'
Code: 'System.ServiceModel.FaultCode'
Message: 'Not enough storage is available to complete this operation.'
Reason: 'Not enough storage is available to complete this operation.'
Data: 'System.Collections.ListDictionaryInternal'
InnerException: ''
TargetSite: 'Void HandleReturnMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage)'
StackTrace: '
Server stack trace:
at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at SolarWinds.Collector.Contract.IPollingControllerService.PollNow(String entityIdentifier)
at SolarWinds.Orion.Core.BusinessLayer.PollingController.<>c__DisplayClass1.<PollNow>b__0(IPollingControllerService n)
at SolarWinds.Orion.Core.Common.ChannelProxy`1.<>c__DisplayClass1.<Invoke>b__0(T n)
at SolarWinds.Orion.Core.Common.ChannelProxy`1.CallMethodWithSharedChannel[TResult](Func`2 fnc)
at SolarWinds.Orion.Core.Common.ChannelProxy`1.Invoke[TResult](Func`2 fnc, ChannelProxyBehavior behavior, String methodName)
at SolarWinds.Orion.Core.Common.ChannelProxy`1.Invoke(Action`1 action, ChannelProxyBehavior behavior)
at SolarWinds.Orion.Core.Common.ChannelProxy`1.Invoke(Action`1 action)
at SolarWinds.Orion.Core.BusinessLayer.PollingController.PollNow(String entityIdentifier)
at SolarWinds.Orion.Core.BusinessLayer.CoreBusinessLayerService.PollNow(String netObjectId)
at SyncInvokePollNow(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)'
HelpLink: ''
Source: 'mscorlib'