Monitoring Tomcat – MBean Metrics Part I

In this blog post I will show you how to use the JavaApp to monitor application servers like Tomcat, JBoss and FUSE.
We’ll start off with Tomcat, since it is fairly easy to set up.

Before we get started, let’s take a look at the steps.

1. Select Tomcat JMX Attributes.
2. Download and Configure the JavaApp
3. Create a Custom Data Collection Service
4. Start Monitoring

 

When you run a java application you have a variety of ways of monitoring the performance and health of your application. You can

• inspect the application logs
• monitor the Jmx MBeans
• inspect GC logs

We are interested in seeing the performance profile of our tomcat server. We will start by browsing the jmx attributes using jconsole to select a group of metrics that we are interested in tracking. Then we will build a service to collect this data and chart it as it produced.

Here’s a check-list of tools and components that will help you along

• Logscape Installation  (Download)
• JavaApp-1.1.zip  (Download the JavaApp)
• JavaApp-1.1.properties (Get a sample properties file)
• Jconsole or JVisualVM

 

Let”s get started!

Introduction

Start JConsole and point it to your tomcat instance. On my test server I am running Ubuntu 12.04 and tomcat7. Ubuntu stores the JAVA_OPTS settings in this file

/etc/defaults/tomcat7

I updated my environment with these JAVA_OPTS settings

JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.port=9000"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.authenticate=false"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.ssl=false"

See the official tomcat7 documentation on how to configure tomcat for jmx monitoring and then restart your tomcat session

sudo service tomcat7 restart

The next step is to connect using JConsole.

Connect the JavaApp to Tomcat

Using Jconsole connect to tomcat.   I am using a stock tomcat installation so my  jmx url is the default. It may differ depending on your environment.

service:jmx:rmi:///jndi/rmi://10.28.1.164:9000/jmxrmi

Now that you are connected take a look at what MBeans are available. We have the standard java MBeans which give us HeapMemory, GC Metrics and runtime information under the java.lang  jmx folder.  Tomcat provides a lot of useful information under the Catalina folder.

轉存失敗重新上傳取消

1.) Select Tomcat JMX Attributes.

There is the potential to write a full-blown tomcat monitoring app but in the scope of this blog post we”ll take a look at the basic MBean attributes for monitoring.

 

Mbean	Attributes	Object Name
ThreadPool	currentThreadCount , currentThreadsBusy, maxConnections	Catalina:type=ThreadPool,name=”http-bio-8888″
RequestProcessor	bytesSent, errorCount, requestCount	Catalina:type=GlobalRequestProcessor,name=”http-bio-8888″
ServletMetrics	errorCount, requestCount, maxTime, minTime	Catalina:j2eeType=Servlet,name=*,WebModule=*,J2EEApplication=none,J2EEServer=none

---

The MBeans we are interested in are the ThreadPools, RequestProcessor, Servlet and the Cache. This is a good place to start as any.

2.) Download and Configure the JavaApp

The JavaApp is designed to monitor all aspects of the jvm from garbage collection, memory pools to the application logs. We need the following files to continue.

• JavaApp-1.0.zip
• JavaApp-1.0-overrides.properties

The properties file allows us to configure the JavaApp.  Read how to add remote jvms to see how remote java processes can be added for monitoring. My properties file looks like this:

*.tomcat.ports=9000
*.tomcat.hosts=10.28.1.164
*.tomcat.label=appserver

I have also decided to add my logscape agents (running jmx on port 8989) to this configuration too. So these lines are also included

*.logscape.ports=8989
*.logscape.hosts=10.28.1.164,10.28.1.160,10.28.1.159,10.28.1.163
*.logscape.label=ll_
*.dashboard.ports=8998
*.dashboard.hosts=10.28.1.160
*.dashboard.label=ll_dashboard

This properties file instructs the JavaApp to to connect to the JMX port 9000 on host 10.28.1.164 and then to label each event with appserver. The label feature of the JavaApp is useful when you have hundreds of jvms across multiple hosts. Once you are happy with this you are ready to deploy the JavaApp-1.0 and see the default JVM MBean metrics.

轉存失敗重新上傳取消

In a future post, we’ll  explore the JavaApp in more detail. After a few minutes you should be getting the standard jvm metrics such cpu and memory. Now that we have this place, it”s time to create a custom JavaApp service to collect our Tomcat metrics.

3.) Writing a Custom Service

Unzip the JavaApp-1.0.zip and edit the bundle file. The bundle file contains a Service descriptor for the data collection services. We are going to create our own to collect tomcat metrics. To create a new service use the following template:

<Service>
<name>HeapJmx</name>
<resourceSelection>type containsAny Manager</resourceSelection>
<fork>true</fork>
<background>true</background>
<instanceCount>-1</instanceCount>
<pauseSeconds>3600</pauseSeconds>
<script>javaapp.groovy query=java.lang:type=Memory@HeapMemoryUsage@NonHeapMemoryUsage  joinQuery=java.lang:type=Runtime@Name pollInterval=60 pollCount=60 </script>
</Service>

Let’s take some time to look at this. The name tag is the name of the Service but is also the name of the file containing the output when it executes. The resourceSelection , fork , background , instanceCount can be left at these values for most installations. The pauseSeconds tag says how long the service will run before being restarted. We’ve set this to an hour. The script tag can be

• A groovy script
• batchf ile or a shell script
• vbs script
• an executable

In our JavaApp service we will use the javaaap.groovy script that comes with the JavaApp to query the jmx. Here’s a list of required parameters

• query: OBJECTNAME@ATTR1@ATTR2
• joinQuery: java.lang:type=Runtime@Name
• pollCount: 60 The number of iterations of the service
• pollInterval=60 Number of seconds before polling for more metrics

To create our first service  we will use this jmx query

query=Catalina:type=ThreadPool,name="http-bio-8888"@currentThreadsBusy@currentThreadCount@maxThreads

Since we can have potentially many connectors replace “http-bio-8888” with a * to retrieve metrics from all connectors.

<script>javaapp.groovy query=Catalina:type=ThreadPool,name=*@currentThreadsBusy@currentThreadCount@maxThreads joinQuery=java.lang:type=Runtime@Name pollInterval=60 pollCount=60</script>

and we will name our service ThreadPool. Before we continue let’s test our query using the debug.sh script found in the JavaApp-1.0 folder

./debug.sh $HOST $PORT $QUERY

Using this syntax I now have:

./debug.sh 10.28.1.164 9000 Catalina:type=ThreadPool,name=*@currentThreadsBusy@currentThreadCount@maxThreads

this returns a timestamped json output line

10-Jul-201412:24:11BST {"objectName":"Catalina:type=ThreadPool,name="http-bio-8888"","currentThreadsBusy":"0","currentThreadCount":"10","maxThreads":"200","__Name":"24260@AGT-LON-UBU3","namespace":"_.*.group1","label":"ll_","host":"10.28.1.164"}