3.1 Installing OPENMARU Cloud APM System Agent

Download System Agent

Log in to the console of the OPENMARU APM server and download the file for installation.

When you select Agent Installation Guide > SYS AGENT, the page where you can click Download SYS Agent is displayed as shown below.

image

Download the file to install the system agent.

If you are using a Linux operating system, you can download it directly using the wget command as shown below.

  • Download the system agent.

$ wget https://cloud.openmaru.io/api/v1/projects/[Project-ID]/download/sys-agent.zip

Install the system agent

  • extract the file.

$ unzip sys-agent.zip
  • change agent settings.

$ cd khan-sys-agent
$ vi conf/khan-sys-agent.conf
  • khan-sys-agent.conf

khan.host=cDIwMjEwODAwMDYuYXBtLmEtYXBuZTIub3Blbm1hcnUuaW8=

khan.port=443

# specify this machine's ip address, only valid ip address of this machine can be used.
#khan.agent.ip=192.168.10.11

# agent's compression algorithm : snappy(default), lzw
#agent.compress.type=lzw

# Gathering netstat info : true(default)
#khan.netstat=false

# include network interface
# configuration sample for openshift
#khan.network.interfaces=eth0,eth1,ens192,lo,tun0,vxlan_sys_4789
# configuration sample for kubernetes
#khan.network.interfaces=eth0,eth1,ens192,lo,tun0,kube-ipvs0,vxlan.calico,nodelocaldns

# include disk mount
# configuration sample for openshift
#khan.disk.mounts=/,/home,/data,/var/lib/docker/containers,C,D

# Allow command list
#khan.allow.commands=service,systemctl

# Memory sigar supported OS vendor version : Enterprise Linux 5,5,Enterprise Linux 6,6(default) startsWith
#khan.memory.sigar.supported.os.vendor.versions=Enterprise Linux 5,5,Enterprise Linux 6,6

Settings in the khan-sys-agent.conf file

The settings in the khan-sys-agent.conf file are as follows.

Settings (environment variables) Description Default

khan.host
(OMAPM_HOST)

Enter the IP of the OPENMARU APM server.
or http:// or ws:// for domain (https:// or wss:// for TLS)

khan.port

khan.port
(OMAPM_PORT)

Enter the port number of the OPENMARU APM server.

khan.tls
(OMAPM_TLS)

TLS enabled or not

false

agent.compress.type
(omappm_agent_compress_type)

Encryption algorithm

snappy

khan.netstat
(OMAPM_NETSTAT)

Whether to monitor network (netstat)

true

khan.network.interfaces
(omapm_network_interfaces)

Monitor only set network interfaces + (OMAPM_NETWORK_INTERFACES) ex) eth0,eth1,ens192,lo,tun0,vxlan_sys_4789

All interfaces

khan.disk.mounts
(omapm_disk_mounts)

Monitor only set mount disks + (OMAPM_DISK_MOUNTS) ex) /,/home,/data,/var/lib/docker/containers,C,D

khan.allow.commands
(omapm_allow_commands)

Restrict execution nouns in the agent + (OMAPM_ALLOW_COMMANDS) ex) service,systemctl

OMAPM_NAME

SYS agent name Separation when installing more than one agent on one machine (Openmaru SYS Container images only)

Setting up plugins for the system agent

The System Agent can use plugins to extend its functionality.

Currently, APM provides the following plugins: Apache monitoring plugin, nginx monitoring plugin, cubrid monitoring plugin, Docker container monitoring plugin, and CRI-O container monitoring plugin.

Set disabled.txt in the plugins directory and start the system agent to run the plugin module.

Enable a plugin

To enable a plugin, you can set which plugins to use by setting the following in the plugins/disabled.txt file.

  • plugins/disabled.txt file.

#.
# name to disable plugin
# # name to disable plugin

khan-plugin-apache
khan-plugin-nginx
khan-plugin-docker
khan-plugin-crio
khan-plugin-containerd
khan-plugin-cubrid
khan-plugin-mysql
  • Enable the khan-plugin-apache and khan-plugin-docker plugins.

#.
# name to disable plugin
# # name to disable plugin

#khan-plugin-apache
khan-plugin-nginx
#khan-plugin-docker
khan-plugin-crio
khan-plugin-containerd
khan-plugin-cubrid
khan-plugin-mysql

Copy the plugin zip file to the plugins directory, and start the system agent to run the plugin module. At this time, the directory is created by unzipping the zip file.

Apache plugin settings

The Apache plugin is configured using the config.properties file under plugins/khan-plugin-apache-5.1.0/conf.

Since the configuration file can be changed by upgrading the plugin, you can separate the plugin file from the configuration file by setting it in the config.properties file in the plugins/khan-plugin-apache.conf directory.

  • *config.properties configuration file contents

instance=apache
host=localhost
port=80
username=
password=
infoUrlPath=/server-info?server
statusUrlPath=/status?auto
jkStatusUrlPath=/jkstatus
modClusterManagerUrlPath=/mod_cluster_manager

maxConnections=1
maxConnectionsPerHost=1
maxConnectionsPerRoute=1
waitIdleCheck=60000
waitIdleClose=120
requestTimeout=1000
sessionTimeout=1000

## "key":"log directory path":"http status code index" - split with whitespaces, ex) homepage:/data/logs/homepage:8
#http.status.code.log.dir=homepage:/data/logs/homepage:8,admin:/data/logs/admin:8
#http.status.code.log.exclude.url.patterns=\\*/,/status,/server-info,/jkstatus,/mod_cluster_manager
#http.status.code.log.exclude.file.patterns=^httpd.log$,.*mod_jk.*,.*modjk.*,.*error_log.*,.*modcluster.*
Settings Description Defaults

instance

Apache instance name to be displayed in OPENMARU APM

apache

host

Hostname of the Apache server to collect information from, preferably localhost

localhost

port

Port number of the Apache server to collect information from

80

username

Not currently used

password

currently deprecated

infoUrlPath

URL where Apache’s mod_info is set.

Make it accessible from localhost.

statusUrlPath

URL where Apache’s mod_status is set.

Make it accessible from localhost.

jkStatusUrlPath

Apache jkstatus page URL.

/jkstatus

modClusterManagerUrlPath

Enter the URL of the Apache mod cluster manager access page.

/mod_cluster_manager

maxConnections

Maximum number of connections to collect Apache information, 1 is sufficient

maxConnectionsPerHost

The number of connections to collect Apache information.

maxConnectionsPerRoute

Number of connections to collect Apache information

waitIdleCheck

Idle check wait time

waitIdleClose

Idle Close wait time

requestTimeout

Timeout if no response to request

sessionTimeout

Session Timeout

http.status.code.log.dir

HTTP response code monitoring settings

homepage:/data/logs/homepage:8 Service name to be displayed in the console: Web server log directory path: Response code location (space-based)

http.status.code.log.exclude.url.patterns

Specify URL patterns to exclude from response code monitoring

http.status.code.log.exclude.file.patterns

Specify log file patterns to exclude from response code monitoring

How to set up Apache for APM monitoring

In the httpd.conf file, set the permissions to read /status, /server-info URLs from localhost as shown below. At this time, set ExtendedStatus to On to get more information from server-status.

#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
ExtendedStatus On

#
# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Change the ".example.com" to match your domain to enable.
#
<Location /status>
  SetHandler server-status
  Order deny,allow
  Deny from all
  Allow from localhost
</Location>

#
# Allow remote server configuration reports, with the URL of
# http://servername/server-info (requires that mod_info.c be loaded).
# Change the ".example.com" to match your domain to enable.
#
<Location /server-info>
  SetHandler server-info
  Order deny,allow
  Deny from all
  Allow from localhost
</Location>

NGINX plugin settings

The nginx plugin configuration uses the config.properties file located under plugins/khan-plugin-nginx-5.1.0/conf. Since the configuration file can be changed by upgrading the plugin, you can separate the plugin file from the configuration file by setting it in the config.properties file in the plugins/khan-plugin-nginx.conf directory.

  • config.properties configuration file contents.

#.
# Nginx status monitoring
# # Nginx status monitoring

instance=nginx
host=localhost
port=80
username=
password=
statusUrlPath=/nginx_status

maxConnections=1
maxConnectionsPerHost=1
maxConnectionsPerRoute=1
waitIdleCheck=60000
waitIdleClose=120
requestTimeout=1000
sessionTimeout=1000
Settings Description Default

instance

Nginx instance name to be displayed in OPENMARU APM

nginx

host

Hostname of the nginx server to collect information from, preferably localhost

localhost

port

Port number of the nginx server to collect information from

80

username

Not currently used

password

Deprecated

statusUrlPath

URL where nginx’s http_stub_status_module is set to

Make it accessible from localhost.

maxConnections

Maximum number of connections to collect nginx information, 1 is sufficient

1

maxConnectionsPerHost

Number of connections to collect nginx information

maxConnectionsPerRoute

Number of connections to collect nginx information

waitIdleCheck

Idle check wait time

waitIdleClose

Idle Close wait time

requestTimeout

Timeout if there is no response to a request

sessionTimeout

Session Timeout

How to set up nginx for APM monitoring

Since ngx_http_stub_status_module is not built by default, add the following to the nginx/conf.d/default.conf file.

    location /nginx_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
}

Cubrid Plugin Settings

The configuration of the cubrid plugin uses the config.properties file under plugins/khan-plugin-cubrid-5.1.0/conf. Since the configuration file can be changed by upgrading the plugin, you can separate the plugin file from the configuration file by setting it in the config.properties file in the plugins/khan-plugin-cubrid.conf directory.

  • config.properties configuration file contents.

#.
# Cubrid status monitoring
# # Cubrid status monitoring

# cubrid server home directory
# if cubridHome is empty then check environment variable 'CUBRID'
cubridHome=

# comma separated broker names to exclude gather stats
excludedBrokers=query_editor
Settings Description Default

cubridHome

Enter the path to the home directory of cubrid.

excludeBrokers

Enter the name of the broker to exclude from status collection.

How to set up Cubrid for APM monitoring

In the cubrid home directory, modify conf/cm.conf support_mon_statistic to YES to enable resource monitoring of the database.

~
# support monitoring statistic (YES/NO), default NO
# If you want to use the advanced monitoring feature on CUBRID Manager,
# you should set this value to YES.

support_mon_statistic=YES
~

APM DASHBOARD settings

In APM dashborad Home Settings > System Settings > System In Plugin Options, set Enable CUBRID DB Monitoring to TRUE and save it.

image

Docker plugin settings

To configure the Docker plugin, use the config.properties file under plugins/khan-plugin-docker-5.1.0/conf.

Since the configuration file can be changed by upgrading the plugin, you can separate the plugin file from the configuration file by setting it in the config.properties file in the plugins/khan-plugin-docker.conf directory.

  • config.properties configuration file contents.

#
# Docker status monitoring
#

# Add docker remote api options in /etc/sysconfig/docker file.
# OPTIONS='...', -H tcp://0.0.0.0:4243 or -H tcp://127.0.0.1:4243
#docker.host=localhost
#docker.port=4243

# docker daemon connect type : sock or net
# Must use sock in Openshift
docker.connect.type=sock

docker.sock=/var/run/docker.sock

# docker's cgroup driver type : cgroup or systemd
# Openshift 3.6 use systemd type, below Openshift 3.5 use cgroup type
docker.cgroup.driver.type=systemd
#docker.cgroup.driver.type=cgroup


# connect timeout in millisecond
docker.connectTimeout=5000

# read timeout in millisecond
docker.readTimeout=2000
Settings Description Default

docker.host

Enter the hostname or IP of the docker daemon.

localhost

docker.port

Enter the port number of the docker daemon.

4342

docker.connect.type

Specify whether you want to use a socket file generated by Docker Engine or access Docker Engine using HTTP.

You can use sock or net.

On Openshift, you must use sock.

sock

docker.sock

Specifies the socket file that Docker Engine creates. On Red Hat flavors of Linux, use the /var/run/docker.sock file.

/var/run/docker.sock

docker.cgroup.driver.type

Specifies the cgroup driver type. Openshift 3.6 uses systemd and Openshift 3.5 uses cgroup.

CRI-O plugin settings

The CRI-O plugin is configured using the config.properties file under plugins/khan-plugin-crio-5.1.0/conf.

Since the configuration file can be changed by upgrading the plugin, you can separate the plugin file from the configuration file by setting it in the config.properties file in the plugins/khan-plugin-crio.conf directory.

  • +] config.properties configuration file contents.

#
# CRI-O Container status monitoring
#

# crio sock file location
crio.sock=/var/run/crio/crio.sock

# connect timeout in millisecond
crio.connectTimeout=5000

# read timeout in millisecond
crio.readTimeout=2000
Settings Description Defaults

crio.sock

Specifies the socket file that CRI-O creates. On Red Hat-like Linux, the file /var/run/crio.sock is used.

/var/run/crio/crio.sock

CONTAINERD plugin settings

The Containerd plugin is configured using the config.properties file under plugins/khan-plugin-containerd-5.1.0/conf.

Since the configuration file can be changed by upgrading the plugin, you can separate the plugin file from the configuration file by setting it in the config.properties file in the plugins/khan-plugin-crio.conf directory.

  • config.properties configuration file contents

#
# khan-plugin-containerd
#

# containerd sock file location
containerd.sock=/var/run/containerd/containerd.sock
# connect timeout in millisecond
containerd.connectTimeout=5000
# read timeout in millisecond
containerd.readTimeout=2000
Settings Description Default

containerd.sock

Specifies the socket file that CONTAINERD creates. On Red Hat-like Linux, use the /var/run/containerd/containerd.sock file.

/var/run/containerd/containerd.sock

MySQL plugin settings

The configuration of the Containerd plugin uses the config.properties file under plugins/khan-plugin-mysql.conf.

Since the configuration file can be changed by upgrading the plugin, you can separate the plugin file from the configuration file by setting it in the config.properties file in the plugins/khan-plugin-mysql.conf directory.

The MySQL plugin supports Mysql, Mariadb, and Percona.

Before setting up, you need to create a MySQL account and assign permissions.

For MySQL 5.6 or MySQL 5.7, use the following command to create a user.

## Create an account
mysql> CREATE USER 'openmaru'@'localhost' IDENTIFIED BY '[PASSWORD]';

For MySQL 8.0 and later, create a user using the password encryption method.

mysql> CREATE USER 'openmaru'@'localhost' IDENTIFIED WITH mysql_native_password by '[PASSWORD]';

Register the privileges.

## Get a list of running sessions
mysql> GRANT PROCESS,SELECT ON *.* TO 'openmaru'@'localhost';

## Force query termination on a running session
mysql> grant super on *.* to 'openmaru'@'localhost';

If performance_schema is not enabled

mysql> SHOW VARIABLES LIKE 'performance_schema';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| performance_schema | ON |
+--------------------+-------+

mysql> GRANT SELECT ON performance_schema.* TO 'openmaru'@'localhost';
  • Enable Slow Log monitoring

Only available when log_output is TABLE
mysql> show global variables where VARIABLE_NAME in ('slow_query_log', 'log_output');
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| slow_query_log | ON |
| log_output | TABLE |
+--------------------+-------+

my.conf
[mysqld]
general_log=on
log_output='TABLE'
  • *config.properties configuration file contents

#
# khan-plugin-mysql
#

# Enable remote monitoring (default: false)
#0.remote.enabled=false
0.cluster.name=
0.instance.id=om-mariadb
#  - ex) jdbc:mariadb://localhost:3306/mysql, jdbc:mysql://localhost:3306/mysql
0.jdbc.jdbcUrl=jdbc:mariadb://192.168.23.23:3306/mysql
#  - ex) org.mariadb.jdbc.Driver, com.mysql.jdbc.Driver, com.mysql.cj.jdbc.Driver
0.jdbc.driverClassName=org.mariadb.jdbc.Driver
0.jdbc.username=root
0.jdbc.password=opennaru
0.jdbc.maximumPoolSize=10
0.jdbc.minimumIdle=0
0.jdbc.validationTimeout=5000
0.jdbc.connectionTimeout=5000
0.path.my.cnf=/etc/my.cnf
  • The jdbc settings follow the HikariCP options.

  • For numeric prefixes, such as 0., utilize it when monitoring N MYSQLs (not only local but also remote MYSQLs can be monitored).

    • 0.instance.id=db1

    • 0…​.

    • 1.instance.id=db2

    • 1…​.

Setting Description Default

0.cluster.name

If HA is configured, recognize as a group if the name is the same

0.instance.id

A distinguishable name for the collection server

Required

0.jdbc.jdbcUrl

Connection JDBC URL

Required

0.jdbc.driverClassName

Driver Class

Required, org.mariadb.jdbc.Driver, com.mysql.cj.jdbc.Driver

0.jdbc.username

User Name

Required

0.jdbc.password

Password

Required

0.jdbc.maximumPoolSize

Pool maximum size

10

0.jdbc.minimumIdle

Pool minimum size

0

0.jdbc.validationTimeout

Validation check timeout

5000

0.jdbc.connectionTimeout

Time to wait before fetching the pool

0.5 seconds

0.path.my.cnf

my.conf configuration file location + (if monitoring locally) (for local monitoring)

/etc/my.cnf

  • +] JDBC driver

Replace the driver files in the following directory if they are not compatible.

  • plugins/khan-plugin-mysql-5.1.0/lib/

    • mysql-connector-java-8.0.30.jar

    • mariadb-java-client-1.8.0.jar

Starting and ending the system agent

  • start the system agent.

$ cd khan-sys-agent/bin
$ ./start.sh
  • *shut down system agent

$ ./kill.sh
  • *Check system agent logs

$ ./tail.sh
2021-08-18 13:43:03 [INFO ] KhanAgentConfig.java::setPluginsStatus:371 - plugin khan-plugin-apache = status: enable
2021-08-18 13:43:03 [INFO ] KhanAgentConfig.java::setPluginsStatus:371 - plugin khan-plugin-docker = status: disable
2021-08-18 13:43:03 [INFO ] KhanAgentConfig.java::setPluginsStatus:371 - plugin khan-plugin-cubrid = status: enable
2021-08-18 13:43:03 [INFO ] KhanAgentConfig.java::setPluginsStatus:371 - plugin khan-plugin-service-checker = status: enable
2021-08-18 13:43:03 [INFO ] KhanAgentConfig.java::setPluginsStatus:371 - plugin khan-plugin-nginx = status: disable
2021-08-18 13:43:03 [INFO ] KhanAgentConfig.java::setPluginsStatus:371 - plugin khan-plugin-crio = status: disable
2021-08-18 13:43:03 [INFO ] ApacheStatusPlugin.java::start:64 - Starting Apache Plugin
2021-08-18 13:43:03 [INFO ] CubridStatusPlugin.java::start:37 - Starting Cubrid Plugin
2021-08-18 13:43:03 [INFO ] ServiceCheckerPlugin.java::start:44 - Starting Service Checker Plugin
2021-08-18 13:43:03 [INFO ] WebSocketService.java::startService:70 - Server URL : wss://p2021080005.apm.a-apne2.openmaru.io:443/monitoring/websocket
2021-08-18 13:43:03 [INFO ] WebSocketClient.java::createBootstrap:129 - Trying to connect : wss://p2021080005.apm.a-apne2.openmaru.io:443/monitoring/websocket
2021-08-18 13:43:04 [INFO ] WebSocketClient.java::operationComplete:140 - Connected.
2021-08-18 13:43:08 [INFO ] WebSocketClientHandler.java::channelRead0:115 - Successfully Connected.