Let's say that you want to monitor a custom service written in Java:
root@myserver # ps ax|grep mycustomserver
8945 ? Sl 11:04 /opt/mycustomserver/server -JVMargs /opt/mycustomserver/server.ini -n mycustomserver
14810 pts/2 S+ 0:00 grep mycustomserver
You can do it by means of JMX, setting JVM's jmxremote parameters as arguments or in a configuration file. You have to set at least jmxremote.port, there is no default value for it:
root@myserver # cat /opt/mycustomserver/server.ini|grep jmxremote
And if you don't know the application but want to try to monitor it, you can try as root to guess the monitoring port number with netstat:
root@myserver # netstat -tuanp|grep 8945|grep LISTEN
tcp 0 0 0.0.0.0:2055 0.0.0.0:* LISTEN 8945/mycustomserver
tcp 0 0 127.0.0.1:1852 0.0.0.0:* LISTEN 8945/mycustomserver
With the remote port number, and in case authentication is turned on with the corresponding user and password, you can add your Java application to Grid Control clicking in the agent link in Targets -> All Targets tab, and in the next page select JVM from the Add drop-down menu. In the Monitoring Configuration page you have to set at least the Machine name (would be 127.0.0.1) and the Admin Port number fields, and click the OK button.
After adding your JVM properly to Oracle Enterprise Manager, you will be able to check in the All Metrics link a lot of interesting metrics, at least interesting if you have knowledge of Java.
Monitoring and Management Using JMX
Monitoring and Management Using JMX Technology