Setting up Zabbix monitoring for JBoss EAP 6 turns out te be quite different as for JBoss EAP 5. This is because the method of fetching JMX data in EAP 5 is no longer available in EAP 6.
In this post I’ll describe a method on how to enable JMX monitoring for JBoss EAP 5 and JBoss EAP 6 within the same Zabbix installation.
Update 22-01-2014: You can download a custom zabbix-java-gateway RPM file for Zabbix 2.2.1 on RHEL/CentOS6 64-bit from my site here. The process of creating the RPM can be found here.
Please note: You still need to add the remoting-libraries from your EAP installation to /usr/sbin/zabbix_java/lib if you use my RPM.
Problem:
JBoss EAP 5 supports JMX monitoring using RMI, where JBoss EAP 6 does not. EAP 6 uses “remoting-jmx” instead of “rmi”. This is a problem because the Zabbix Java Gateway only supports RMI for the JMX interface, preventing us from monitoring JBoss EAP 6 using JMX monitoring.
Goal:
My goal in this blogpost is to enable JMX monitoring for JBoss EAP 5 and JBoss EAP 6 on the same Zabbix Server.
Steps:
1.Modify source code of the Zabbix Java Gateway to enable both RMI and remoting-jmx
2.Add the required libraries to the Zabbix Java Gateway
3.Enable remote management in JBoss EAP 6
4.Configure a JBoss EAP 5 and JBoss EAP 6 host to test the new configuration
5.Using JConsole to identify items you want to monitor in Zabbix
Let’s get started.
1. Modify source code of the Zabbix Java Gateway to enable both RMI and remoting-jmx
Like stated before, the Zabbix Java Gateway component only supports JMX over RMI. This is because the service URL it is using to connect to the remote server is hardcoded in JMXItemChecker.java:
url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi");
At the time of writing the most current version of Zabbix is 2.2.1, and there is no way to customise this JMX url. JBoss EAP 6 doesn’t support RMI and expects remoting-jmx.
Replacing the url with this one will enable remoting-jmx, at the cost of losing RMI monitoring:
url = new JMXServiceURL("service:jmx:remoting-jmx://" + conn + ":" + port);
In my environment I want to monitor both JBoss EAP 5 and EAP 6, and I don’t want to add a second zabbix server to do so. So while searching for a way to do this with minimal changes to the zabbix source code I had the idea of using a different port number for different JBoss versions.
It’s far from ideal and just a workaround until the Zabbix people come up with a real solution, but it requires only a tiny bit of modifying the source code.
Here are my notes of modifying the source code and building new binaries on a clean CentOS 6.5 machine.
Preparing the build server:
# yum -y update
# reboot
# yum groupinstall "Development Tools"
# yum -y install mysql-server httpd php mysql-devel php-mysql libxml2-devel libcurl-devel php-mbstring php-bcmath php-gd php-xml
Getting the software:
# cd /opt
# wget http://sourceforge.net/projects/zabbix/files/ZABBIX%20Latest%20Stable/2.2.1/zabbix-2.2.1.tar.gz/download
# tar -xvzf zabbix-2.2.1.tar.gz
Modify the JMXItemChecker.java:
# vi zabbix-2.2.1/src/zabbix_java/src/com/zabbix/gateway/JMXItemChecker.java
Change:
1 String conn = request.getString(JSON_TAG_CONN);
2 int port = request.getInt(JSON_TAG_PORT);
3
4 url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi");
5 jmxc = null;
6 mbsc = null;
view raw JMXItemChecker.java hosted with by GitHub
To:
1 String conn = request.getString(JSON_TAG_CONN);
2 int port = request.getInt(JSON_TAG_PORT);
3
4 //Dirty solution for ZBXNEXT-1274
5 Integer remoting = new Integer("7777");
6 int retval = remoting.compareTo(port);
7
8 if (retval == 0)
9 {
10 url = new JMXServiceURL("service:jmx:remoting-jmx://" + conn + ":" + port);
11 }
12 else
13 {
14 url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi");
15 }
16 jmxc = null;
17 mbsc = null;
view raw JMXItemChecker.java hosted with by GitHub
This enables remoting-jmx for hosts with a JMX Interface on port 7777 only, while keeping RMI for any other port. Of course you can change this port to be whatever you need, just be sure to match the port you configure in JBoss.
That is all there is to it, let’s compile the binaries and install our modified version of Zabbix:
# cd zabbix-2.2.1
# ./configure --enable-server --enable-agent --with-mysql --with-libcurl --with-libxml2 --enable-java
# make install
Configure the Java Pollers:
# vi /usr/local/etc/zabbix_server.conf
Add these values and save:
JavaGateway=10.37.129.2
StartJavaPollers=5
JavaGatewayPort=10052
3. Add the required libraries to the Zabbix Java Gateway
Before we can start the java gateway we need to tell it how to use remoting-jmx. We can use the jar files from our JBoss EAP 6 installation to do this.
Find the following jar files in your <EAP_6_HOME>/modules/system/layers/base/<sub_dirs>:
$ cat needed_modules.txt
jboss-as-remoting
jboss-logging
jboss-logmanager
jboss-marshalling
jboss-marshalling-river
jboss-remoting
jboss-sasl
jcl-over-slf4j
jul-to-slf4j-stub
log4j-jboss-logmanager
remoting-jmx
slf4j-api
xnio-api
xnio-nio
Place these in the java gateway lib directory:
$ for i in $(cat needed_modules.txt); do find /opt/jboss-eap-6.0 -iname ${i}*.jar -exec cp {} /usr/local/sbin/zabbix_java/lib/ \; ; done
If you’re reading this post I assume setting up the Zabbix database and web frontend is trivial so I won’t cover it here. Let’s start the java gateway and zabbix server and add a JBoss EAP 5 and JBoss EAP 6 host to test our modifications.
3. Enable remote management in JBoss EAP 6
I’m using a .zip deployment of JBoss EAP 6, unzipped in /opt/jboss-eap-6.0 and running on a host with an external IP number of 10.37.129.5.
In this version of JBoss a user is required to do remote monitoring. Fortunately JBoss comes with the add-user.sh script to do this for you:
[root@jboss6 bin]# cd /opt/jboss-eap-6.0/bin/
[root@jboss6 bin]# ./add-user.sh
What type of user do you wish to add?
a) Management User (mgmt-users.properties)
b) Application User (application-users.properties)
(a):
Enter the details of the new user to add.
Realm (ManagementRealm) :
Username : zabbix
Password : zabbix_1
Re-enter Password : zabbix_1
About to add user 'zabbix' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'zabbix' to file '/opt/jboss-eap-6.0/standalone/configuration/mgmt-users.properties'
Added user 'zabbix' to file '/opt/jboss-eap-6.0/domain/configuration/mgmt-users.properties'
Is this new user going to be used for one AS process to connect to another AS process?
e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.
yes/no? no
We can now use the user “zabbix” with a password of “zabbix_1” for remote management. We also need to configure the management interface of JBoss to listen on port 7777.
I’m going to run my JBoss instance in standalone mode with the standalone-full profile. To configure the management port, open:
/opt/jboss-eap-6.0/standalone/configuration/standalone-full.xml
Search for this line:
<socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>
Change the port number to 7777:
<socket-binding name="management-native" interface="management" port="${jboss.management.native.port:7777}"/>
Save the file and start the JBoss instance:
/opt/jboss-eap-6.0/bin/standalone.sh --server-config=standalone-full.xml -b 10.37.129.5 -Djboss.bind.address.management=10.37.129.5
4. Configure a JBoss EAP 5 and JBoss EAP 6 host to test the new configuration
In the Zabbix web interface go to: Configuration -> Hosts -> Create host.
The first host I will add is a JBoss EAP 5 host, so it will require JMX RMI monitoring:
Hostname: jboss.kanbier.lan
Groups: Linux servers
JMX Interfaces: jboss.kanbier.lan DNS 9999
5. Using JConsole to identify items you want to monitor in Zabbix
There are at least 2 ways to get the JMX keys fron JBoss so you can add them to Zabbix. In my example for JBoss EAP 6 I use the key:
"jboss.as:subsystem=datasources,data-source=ExampleDS","jndiName"
This fetches the “jndiName” from the ExampleDS data-source. How to construct this key? You can easily browse the items that are available for monitoring using either JConsole or jboss-cli.sh. In this example I’ll be using JConsole.
JConsole is a Java GUI application that connects to the management interface of JBoss in very much the same way as the Zabbix Java Gateway. It is shipped with JBoss and located in <JBOSS_HOME>/bin.
Before you start it, make sure your JBoss EAP 6 instance is running using the configuration described in the instructions above.
These are the steps I took to get it started on jboss6.kanbier.lan with external ip 10.37.129.5.
$ export PATH=/usr/java/latest/bin:$PATH $ export JAVA_HOME=/usr/java/latest $ /opt/jboss-eap-6.0/bin/jconsole.sh
This should start a GUI if you have an X session available. Fill in:
Check: Remote Process service:jmx:remoting-jmx://10.37.129.5:7777 Username: zabbix Password: zabbix_1
But there is one downside, the data available is only realtime. So no history unless you keep JConsole running forever. This is where Zabbix comes in. You can pull the data from the management interface just like JConsole does and store it in Zabbix so you can build up a history, analyse trends and all things Zabbix does best.
Let’s say you want to monitor the HeapMemoryUsage in Zabbix. This requires some knowledge of JBoss, but here is how to find it in JConsole.
Click on the MBeans tab -> java.lang -> Memory -> Attributes -> HeapMemoryUsage: