Hitachi Compute Blade (BladeSymphony) Performance Monitoring

Installation

Hitachi Compute Blade (BladeSymphony)

Skip Prerequisites, Web and LPAR2RRD tabs in case of configuring Virtual Appliance

Prerequisites
Web
LPAR2RRD
Hitachi Compute Blade
Linux OS agent

LPAR2RRD can be hosted on any Unix or Linux Operating system
Perl: arbitrary version coming with the OS with some additional modules, more below
Apache or any other Web server

Skip it in case of configuring Virtual Appliance
HW sizing

User creation

create lpar2rrd user under root:

# useradd -c "LPAR2RRD user" -m lpar2rrd

Increase limits for user lpar2rrd and the WEB server user (under root)

AIX

# chuser  nofiles=32768 data=-1  stack=-1 rss=-1 lpar2rrd
# chdev -l sys0 -a maxuproc=2000
# chuser  data=-1  stack=-1 rss=-1 apache

Linux

# vi /etc/security/limits.conf

lpar2rrd        hard    stack           -1
lpar2rrd        soft    stack           -1
lpar2rrd        hard    data            -1
lpar2rrd        soft    data            -1
lpar2rrd        hard    nofile          32768 
lpar2rrd        soft    nofile          32768 
lpar2rrd        hard    nproc           5000
lpar2rrd        soft    nproc           5000

apache          hard    stack           -1
apache          soft    stack           -1
apache          hard    data            -1
apache          soft    data            -1

If you have already running STOR2RRD on the same host then skip to Web tab.

Linux RedHat, CentOS

# ARCH=$( /bin/arch )
# subscription-manager repos --enable "codeready-builder-for-rhel-8-${ARCH}-rpms"

# yum config-manager --set-enabled PowerTools

# umask 0022 
# yum install perl rrdtool rrdtool-perl httpd
# yum install epel-release
# yum install perl-TimeDate perl-XML-Simple perl-XML-SAX perl-XML-LibXML perl-Env perl-CGI perl-Data-Dumper perl-LWP-Protocol-https perl-libwww-perl perl-Time-HiRes perl-IO-Tty
# yum install perl-PDF-API2
# yum install ed bc libxml2 sharutils

PDF reporting

rrdtool-perl-1.4.8-9.el7.x86_64.rpm

epel-release-7-9.noarch.rpm

Linux Debian, Ubuntu

# umask 0022 
# apt-get install apache2 perl rrdtool ed bc
# apt-get install librrdp-perl libxml-sax-perl libxml-simple-perl libtimedate-perl libenv-sanctify-perl libcgi-pm-perl libdata-dumper-simple-perl libpdf-api2-perl libxml2-utils
# apt-get install liblwp-protocol-https-perl libcrypt-ssleay-perl libio-socket-ssl-perl libmozilla-ldap-perl libxml-parser-perl

AIX

RRDTool 1.4.8

Solaris

# /opt/csw/bin/pkgutil -y -i pm_xml_simple pm_libxml_perl pm_xml_sax rrdtool

Skip it in case of configuring Virtual Appliance

Apache download

Linux CentOS, RedHat
```
# yum install httpd
```
Linux Debian, Ubuntu
```
# apt-get install httpd
```

Apache configuration

Linux CentOS, RedHat
Linux Debian, Ubuntu
Apache 2.4+
Apache 1.3 - 2.3
SELinux disable
Apache authorization (optional)

Apache start / restart

# apachectl restart
# /opt/freeware/apache/sbin/apachectl restart
# /etc/init.d/httpd restart
# service apache2 restart

# firewall-cmd --add-service=http --permanent
# firewall-cmd --add-service=https --permanent
# firewall-cmd --add-port=8162/tcp --permanent

# firewall-cmd --reload

Test web pages

http://<your lpar2rrd host>/lpar2rrd/

Apache configuration per platform and version

Linux CentOS, RedHat

Append at the end of /etc/httpd/conf/httpd.conf following

# vi /etc/httpd/conf/httpd.conf

AddHandler cgi-script .sh
# DocumentRoot  "/home/lpar2rrd/lpar2rrd/www/"
Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
<Directory "/home/lpar2rrd/lpar2rrd/www/">
    AllowOverride AuthConfig
    Options Indexes FollowSymLinks
    Require all granted
</Directory>
# CGI-BIN
ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
<Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
    AllowOverride AuthConfig
    SetHandler cgi-script
    Options ExecCGI FollowSymLinks
    Require all granted
</Directory>

Linux Debian, Ubuntu : Apache2 (2.4+)

Enable CGI-BIN

# a2enmod cgi

Append this at the end of /etc/apache2/apache2.conf

# vi /etc/apache2/apache2.conf

AddHandler cgi-script .sh
Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
<Directory "/home/lpar2rrd/lpar2rrd/www/">
    Options Indexes FollowSymLinks 
    Require all granted
</Directory>
# CGI-BIN
ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
<Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
    AllowOverride None
    Options +ExecCGI +SymLinksIfOwnerMatch
    Require all granted
    SetHandler cgi-script
</Directory>

Apache 2.4+ generally

Configure httpd.conf, it might be in different locations (depends of distribution)

/etc/httpd/httpd.conf
/etc/httpd/conf/httpd.conf
/etc/opt/freeware/apache/httpd.conf (AIX)
/opt/freeware/etc/httpd/httpd.conf (AIX)
/opt/freeware/etc/httpd/conf/httpd.conf (AIX)
/etc/apache2/apache2.conf

Verify alias_module and cgi_module is loaded (uncommented it, the path can be different)

LoadModule alias_module /opt/freeware/lib/httpd/modules/mod_alias.so
LoadModule cgi_module /opt/freeware/lib/httpd/modules/mod_cgi.so

cgi_module could depend on mpm_prefork_module like here.
Allow it as well (uncomment it) and disable mpm_worker_module, mpm_event_module

LoadModule mpm_prefork_module /opt/freeware/lib/httpd/modules/mod_mpm_prefork.so
# LoadModule mpm_worker_module /opt/freeware/lib/httpd/modules/mod_mpm_worker.so
# LoadModule mpm_event_module modules/mod_mpm_event.so
<IfModule mpm_prefork_module>
     LoadModule cgi_module /opt/freeware/lib/httpd/modules/mod_cgi.so
</IfModule>

Append this at the end of httpd.conf

AddHandler cgi-script .sh
Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
<Directory "/home/lpar2rrd/lpar2rrd/www/">
    Options Indexes FollowSymLinks 
    Require all granted
</Directory>
# CGI-BIN
ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
<Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
    AllowOverride None
    SetHandler cgi-script
    Options ExecCGI FollowSymLinks
    Require all granted
</Directory>

Sometimes appears necessary to remove "Require all granted" lines in both directives.

Apache 1.3 - 2.3

Configure httpd.conf, it might be in different locations (depends of distribution)

/etc/httpd/httpd.conf
/etc/httpd/conf/httpd.conf

# vi /etc/opt/freeware/apache/httpd.conf

Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
<Directory "/home/lpar2rrd/lpar2rrd/www/">
    Options Indexes FollowSymLinks 
    Order allow,deny
    Allow from all
</Directory>
# CGI-BIN
ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
<Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
    AllowOverride None
    Options ExecCGI FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

SELinux changes

You need to disable the SELinux service protection to be able to utilize basic web services.
You must do it in case you get "(13)Permission denied:" in Apache error_log.
You have 2 options:

Disable SELinux only for Apache

Step 1. Query for the Boolean value you need to change:
# getsebool -a | grep httpd_dis
  httpd_disable_trans --> off

Step 2. Disable the SELinux protection:
# setsebool -P httpd_disable_trans=1

Step 3. Verify that the Boolean has changed:
# getsebool -a | grep httpd_dis
  httpd_disable_trans --> on

Globally disable SELinux

# setenforce Permissive
# vi /etc/sysconfig/selinux
  SELINUX=disabled

Implementation can be done in 2 scenarios:

Hitachi HvmSh is installed on the LPAR2RRD server (it must be Linux)
Hitachi HvmSh in installed on any Linux server

Install HvmSh API

Download from Hitachi site HVM Management Command package
install it either on LPAR2RRD server (if it is running Linux) or whenever else on a Linux machine
It can be installed only on a Linux machine.
As install directory you can use /home/lpar2rrd/lpar2rrd/HvmShAPI/

Allow access from HvmSh API server to all HVM hosts on ports 623:UDP, 23250:TCP (in case of SSL 20670:TCP)
If you installed it on the LPAR2RRD server then you can check connectivity by:

$ perl /home/lpar2rrd/lpar2rrd/bin/conntest.pl 192.168.1.1 23250
  Connection to "192.168.1.1" on port "23250" is ok
$ perl /home/lpar2rrd/lpar2rrd/bin/conntest_udp.pl 192.168.1.1 623
  Connection to "192.168.1.1" on port "623" is ok

Install SSL certificates to allow HvmSh access your blades if needed (it is not required)
Use Hitachi documentation for that.
Allow IP access by setting: "HVM CLI1 IP Address" through telnet interface

Test connection to your blades, you need to run it twice, first attempt might end up with an error, ignore it

$ /home/lpar2rrd/lpar2rrd/HvmShAPI/HvmSh -host=<HVM host IP/hostname> get HvmPerfMon filename=/tmp/test.txt noconf nodetail exio
$ /home/lpar2rrd/lpar2rrd/HvmShAPI/HvmSh -host=<HVM host IP/hostname> get HvmPerfMon filename=/tmp/test.txt noconf nodetail exio

file /tmp/test.txt will contain performance data from your blade

LPAR2RRD OS agent

Install the latest LPAR2RRD OS agent on a host with HvmSh API installed
Download
Install
If HvmSh API is not directly on LPAR2RRD server then verify then allow network conectivity from HvmSh host to LPAR2RRD server on port 8162
```
$ perl /home/lpar2rrd/lpar2rrd/bin/conntest.pl 192.168.1.2 8162
  Connection to "192.168.1.2" on port "8162" is ok
```

Test agent

/usr/bin/perl /home/lpar2rrd/agent/lpar2rrd-agent.pl -i <HVM host IP/hostname> -b /home/lpar2rrd/lpar2rrd/HvmSh <LPAR2RRD server>

schedulle it in crontab like this:

* * * * * /usr/bin/perl /home/lpar2rrd/agent/lpar2rrd-agent.pl -i 192.168.1.1 -b /home/lpar2rrd/lpar2rrd/HvmSh 192.168.1.2 > /var/tmp/lpar2rrd-agent-hitachi.out 2>&1

You will see your Blade(s) in the UI under Hitachi folder within an hour (Ctrl-F5 in the web browser).

OS agent is add-on feature for monitoring from operating system level.
It is monitoring CPU, memory utilization, paging, LAN and SAN traffic on all adapters.
It requires the OS agent deployment to every monitored VM.
The agent is written in Perl and calls basic OS commands to obtain required statistics like vmstat, iostat.

Additional information about the OS agent:

Prerequisites

Perl
Opened TCP communication between each VM and LPAR2RRD server on port 8162.
Connections are initiated from VM side.
Additional disk space on LPAR2RRD server (about 40MB per each monitored VM)
Create preferable dedicated user lpar2rrd on each VM with minimum rights
```
# useradd -c "LPAR2RRD agent user" -m lpar2rrd
```

OS agent installation (client)

Get the latest OS agent from download page

Linux installation under root

# rpm -Uvh lpar2rrd-agent-5.00-0.noarch.rpm
# rpm -qa|grep lpar2rrd-agent
  lpar2rrd-agent-5.00-0

Solaris x86 installation under root:

# gunzip lpar2rrd-agent-5.00-0.solaris-i86pc.tar.gz
# tar xf lpar2rrd-agent-5.00-0.solaris-i86pc.tar
# pkgadd -d .
  The following packages are available:
  1  lpar2rrd-agent     LPAR2RRD OS agent 5.00
                        (i86pc) 5.00
 ...

Solaris upgrade under root:

# pkgrm lpar2rrd-agent
# pkgadd -d .

Schedule its run every minute from the crontab on every VM.
This line must be placed into lpar2rrd crontab:
```
# su - lpar2rrd
$ crontab -e 
* * * * * /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl <LPAR2RRD-SERVER> > /var/tmp/lpar2rrd-agent.out 2>&1
```
Replace <LPAR2RRD-SERVER> by hostname of your LPAR2RRD server.
You might need to add lpar2rrd user into /var/adm/cron/cron.allow under root user if above "crontab -e" fails.
```
# echo "lpar2rrd" >> /var/adm/cron/cron.allow
```

OS agent data graphs will appear in the UI, use Ctrl-F5 to refresh your web browser

Troubleshooting

Client (agent) side:

Test if communication through the LAN is allowed.

$ telnet  <LPAR2RRD-SERVER> 8162
  Connected to 192.168.1.1   .
  Escape character is '^]'.

This is ok, exit either Ctrl-C or ^].

Check following agent files:
data store: /var/tmp/lpar2rrd-agent-*.txt
error log: /var/tmp/lpar2rrd-agent-*.err
output log: /var/tmp/lpar2rrd-agent.out

run the agent from cmd line:

$ /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <LPAR2RRD-SERVER>
  ...
  Agent send     : yes : forced by -d 
  Agent send slp: sending wait: 4
  OS/HMC agent working for server: <LPAR2RRD-SERVER>
  store file for sending is /var/tmp/lpar2rrd-agent-<LPAR2RRD-SERVER>-lpar2rrd.txt

It means that data has been sent to the server, all is fine
Here is example when the agent is not able to sent data :

$ /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <LPAR2RRD-SERVER>
  ...
  Agent send     : yes : forced by -d 
  Agent send slp: sending wait: 1
  OS/HMC agent working for server: <LPAR2RRD-SERVER>
  store file for sending is /var/tmp/lpar2rrd-agent-<LPAR2RRD-SERVER>-lpar2rrd.txt
  Agent timed out after : 50 seconds /opt/lpar2rrd-agent/lpar2rrd-agent.pl:265

It means that the agent could not contact the server.
Check communication, port, above telnet example, DNS resolution of the server etc.

Server side:

test if the daemon on LPAR2RRD server is running, and checking the logs

$ ps -ef|grep lpar2rrd-daemon
  lpar2rrd 10617010 1 0 Mar 16 - 0:00 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/lpar2rrd-daemon.pl
$ cd /home/lpar2rrd/lpar2rrd
$ tail logs/error.log-daemon
$ tail logs/daemon.out
  new server has been found and registered: Linux (lpar=linuxhost01)
  mkdir : /lpar2rrd/data/Linux/no_hmc/linuxhost01/

It means that new OS agent has been registered from linuxhost01 (Linux stand-alone example)

Test if OS agent data is being stored on the LPAR2RRD server and have actual timestamp:

$ cd /home/lpar2rrd/lpar2rrd
$ ls -l data/<server name>/*/<VM name>/*mmm
  -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/cpu.mmm
  -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/lan-en1.mmm
  -rw-r--r-- 2 lpar2rrd staff 10790264 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/mem.mmm
  -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/pgs.mmm
  -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/san-vscsi0.mmm
  -rw-r--r-- 2 lpar2rrd staff  3597208 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/san_resp-vscsi0.mmm
$ find data -name mem.mmm -exec ls -l {} \;
  ...

In case of a problem check our forum or contact us via support@lpar2rrd.com.
We would need this data for start of troubleshooting.

Notes

OS agent upgrade

Skip it in case of configuring Virtual Appliance

Install LPAR2RRD server

Download the latest LPAR2RRD server
Upgrade your already running LPAR2RRD instance.

Install it:

# su - lpar2rrd
$ tar xvf lpar2rrd-7.XX.tar
$ cd lpar2rrd-7.XX
$ ./install.sh
$ cd /home/lpar2rrd/lpar2rrd

Schedule to run it from lpar2rrd crontab (it might already exist there)
```
$ crontab -l | grep load.sh
$
```
Add if it does not exist as above
```
$ crontab -e

# LPAR2RRD UI
0 * * * * /home/lpar2rrd/lpar2rrd/load.sh > /home/lpar2rrd/lpar2rrd/load.out 2>&1 
```
Assure there is just one such entry in crontab.
You might need to add lpar2rrd user into /var/adm/cron/cron.allow if crontab command fails
Allow it for lpar2rrd user as root user.
```
# echo "lpar2rrd" >> /var/adm/cron/cron.allow
```

Initial start from cmd line:

$ cd /home/lpar2rrd/lpar2rrd
$ ./load.sh

Go to the web UI: http://<your web server>/lpar2rrd/
Use Ctrl-F5 to refresh the web browser cache.