I've been experiencing some problems with the Job Engine v2 on one of my poller servers. Every so often, the number of Jobs Running will stall for up to 30 seconds, causing no other jobs to begin execution. Usually, the scheduler will spawn a new worker process to compensate. But, this stalling repeats and another process is spawned. This can happen a lot in succession and increase the worker processes by double. The number of Jobs Queued finally starts to reduce after a minute or two and the worker processes will end about 10 to 15 minutes later.
This would seem to be appropriate behavior, but it is causing a slow memory consumption on the poller finally resulting in the need to restart the server about every three weeks. The attached image shows the Windows Performance Monitor on some of the Job Engine metrics during a relatively calm spike. I have seem spikes happen such that the entire 15 minutes window shows running jobs stalling.
I need to find out what job is causing the stalling. I'm figuring that the stalling may be due to a network timeout, so the 30 seconds is my clue. I'd like to get a report of jobs and their durations. The 30 seconds ones should be easy to pick out and help me find the true problem.
Many thanks in advance.