Technote (troubleshooting)
Problem(Abstract)
New system is working properly but HMGR warning messages are being logged in the SystemOut.log file.
Symptom
[10/25/05 16:42:27:635 EDT] 0000047a CoordinatorCo W HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 9 seconds.
Cause
The HMGR0152W message is an indication that JVM thread scheduling delays are occurring for this process.
The WebSphere® Application Server high availability manager component contains thread scheduling delay detection logic, that periodically schedules a thread to run and tracks whether the thread was dispatched and run as scheduled. By default, a delay detection thread is scheduled to run every 30 seconds, and will log a HMGR0152W message if it is not run within 5 seconds of the expected schedule. The message will indicate the delay time or time differential between when the thread was expected to get the CPU, and when the thread actually got CPU cycles.
The HMGR0152W message can occur even when plenty of CPU resource is available. There are a number of reasons why the scheduled thread might not have been able to get the CPU in a timely fashion. Some common causes include the following:
The WebSphere® Application Server high availability manager component contains thread scheduling delay detection logic, that periodically schedules a thread to run and tracks whether the thread was dispatched and run as scheduled. By default, a delay detection thread is scheduled to run every 30 seconds, and will log a HMGR0152W message if it is not run within 5 seconds of the expected schedule. The message will indicate the delay time or time differential between when the thread was expected to get the CPU, and when the thread actually got CPU cycles.
The HMGR0152W message can occur even when plenty of CPU resource is available. There are a number of reasons why the scheduled thread might not have been able to get the CPU in a timely fashion. Some common causes include the following:
- The physical memory is overcommitted and paging is occurring.
- The heap size for the process is too small causing garbage collection to run too frequently and/or too long, blocking execution of other threads.
- There might simply be too many threads running in the system, and too much load placed on the machine, which might be indicated by high CPU utilization.
Resolving the problem
The HMGR0152W message is attempting to warn you that a condition is occurring that might lead to instability if it is not corrected. Analysis should be performed to understand why the thread scheduling delays are occurring, and what action(s) should be taken. Some common solutions include the following:
- Adding more physical memory to prevent paging.
- Tuning the JVM memory (heap size) for optimal garbage collection.
- Reducing the overall system load to an acceptable value.
If the HMGR0152W messages do not occur very often, and indicate that the thread scheduling delay is relatively short (for example, < 20 seconds), it is likely that no other errors will occur and the message can safely be ignored.
The high availability manager thread scheduling delay detection is configurable by setting either of the following 2 custom properties.
- IBM_CS_THREAD_SCHED_DETECT_PERIOD determines how often a delay detection thread is scheduled to run. The default value of this parameter is 30 (seconds).
- IBM_CS_THREAD_SCHED_DETECT_ERROR determines how long of a delay should be tolerated before a warning message is logged. By default this value is 5 (seconds).
These properties are scoped to a core group and can be configured as follows:
- In the administrative console, click Servers > Core groups > Core groups settings and then select the core group name.
- Under Additional Properties, click Custom properties > New.
- Enter the property name and desired value.
- Save the changes.
- Restart the server for these changes to take effect.
While it is possible to use the custom properties mentioned above to increase the thread-scheduling-detect-period until the HMGR0152W warning messages no longer occur, this is not recommended. The proper solution is to tune the system to eliminate the thread scheduling delays.