RHV / oVirt / OLVM Monitoring
- LPAR2RRD can be hosted on any Unix or Linux Operating system
- Perl: arbitrary version coming with the OS with some additional modules, more below
- Apache or any other Web server
HW sizing
User creation
- create lpar2rrd user under root:
# useradd -c "LPAR2RRD user" -m lpar2rrd # chmod 755 /home/lpar2rrd
- Increase limits for user lpar2rrd and the WEB server user (under root)
- AIX
# chdev -l sys0 -a maxuproc=2000 # chuser nofiles=32768 data=-1 stack=-1 rss=-1 lpar2rrd # chuser data=-1 stack=-1 rss=-1 apache
- Linux
# vi /etc/security/limits.conf lpar2rrd hard stack -1 lpar2rrd soft stack -1 lpar2rrd hard data -1 lpar2rrd soft data -1 lpar2rrd hard nofile 32768 lpar2rrd soft nofile 32768 lpar2rrd hard nproc 5000 lpar2rrd soft nproc 5000 apache hard stack -1 apache soft stack -1 apache hard data -1 apache soft data -1
- AIX
Linux RedHat, CentOS
-
RHEL 8: enable the codeready-builder-for-rhel-8-*-rpms repository
# ARCH=$( /bin/arch ) # subscription-manager repos --enable "codeready-builder-for-rhel-8-${ARCH}-rpms" -
CentOS 8: enable the PowerTools repository
# yum config-manager --set-enabled PowerTools or # dnf config-manager --set-enabled powertools
You can also just open /etc/yum.repos.d/CentOS-Linux-PowerTools.repo with a text editor and set enabled= to 1 instead of 0
- RedHat, CentOS: all versions
Follow this for installing perl-PDF-API2 on RHEL/CentOS 8
# umask 0022 # yum install perl rrdtool rrdtool-perl httpd mod_ssl # yum install epel-release # yum install perl-TimeDate perl-HTTP-Date perl-Env perl-CGI perl-Data-Dumper perl-LWP-Protocol-https perl-libwww-perl perl-Time-HiRes perl-IO-Tty # yum install perl-PDF-API2 perl-JSON-XS perl-XML-Simple perl-XML-NamespaceSupport perl-URI perl-XML-SAX-Base perl-XML-SAX perl-XML-LibXML # yum install ed bc libxml2 sharutil
Note that rrdtool-perl and epel-release (this is necessary only for PDF reporting) might not be in your RedHat base repository especially for RHEL 6.x and olders.
Install CentOS package instead rrdtool-perl-1.4.8-9.el7.x86_64.rpm and epel-release-7-11.noarch.rpm
Linux Debian, Ubuntu
# umask 0022 # apt-get install apache2 perl rrdtool ed bc # apt-get install librrdp-perl libxml-sax-perl libxml-simple-perl libtimedate-perl libenv-sanctify-perl libcgi-pm-perl libdata-dumper-simple-perl libpdf-api2-perl libxml2-utils # apt-get install liblwp-protocol-https-perl libcrypt-ssleay-perl libio-socket-ssl-perl libmozilla-ldap-perl libxml-parser-perl libjson-xs-perl
SuSE
# zypper install rrdtool perl-rrdtool apache2 # zypper install perl-TimeDate perl-XML-Simple perl-XML-SAX perl-XML-LibXML perl-CGI perl-LWP-Protocol-https perl-libwww-perl perl-IO-Tty # zypper install perl-PDF-API2 perl-JSON-XS # zypper install ed bc libxml2-2 sharutils
AIX
-
Use either AIX yum installation of the latest prerequisite tools or manual installation of older version RRDTool 1.4.8
Solaris
# /opt/csw/bin/pkgutil -y -i pm_xml_simple pm_libxml_perl pm_xml_sax rrdtool
Troubleshooting
-
If you get error durring device connection test or in error log like:
- Missing LWP::Protocol::https
- Connection reset by peer
- SSL negotiation failed
Skip it in case of configuring Virtual Appliance
Optional configuration of authorization in Apache. You can proceed it after Apache access works without authorization prompt as per below cfg.
Optional configuration of authorization in Apache. You can proceed it after Apache access works without authorization prompt as per below cfg.
Apache download
- Linux CentOS, RedHat
# yum install httpd
- Linux Debian, Ubuntu
# apt-get install httpd
Apache configuration
- Linux CentOS, RedHat
- Linux Debian, Ubuntu
- SuSE
- Apache 2.4+
- Apache 1.3 - 2.3
- SELinux disable
- Apache authorization (optional)
Apache start / restart
-
Command depends on Apache version, one of these should work:
# apachectl restart # /opt/freeware/apache/sbin/apachectl restart # /etc/init.d/httpd restart # service apache2 restart
# firewall-cmd --add-service=http --permanent # firewall-cmd --add-service=https --permanent # firewall-cmd --add-port=8162/tcp --permanent # firewall-cmd --reload
Test web pages
-
Point your web browser to
http://<your lpar2rrd host>/lpar2rrd/ as soon as you configure LPAR2RRD
Do not forget to ensure that Apache starts after the OS reboot (there must be a starting script in /etc/rc.... )
Apache configuration per platform and version
-
Linux CentOS, RedHat
Append at the end of /etc/httpd/conf/httpd.conf following# vi /etc/httpd/conf/httpd.conf AddHandler cgi-script .sh # DocumentRoot "/home/lpar2rrd/lpar2rrd/www/" Alias /lpar2rrd "/home/lpar2rrd/lpar2rrd/www/" <Directory "/home/lpar2rrd/lpar2rrd/www/"> AllowOverride Options Indexes FollowSymLinks Require all granted </Directory> # CGI-BIN ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/" <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi"> AllowOverride SetHandler cgi-script Options ExecCGI FollowSymLinks Require all granted </Directory> -
Linux Debian, Ubuntu : Apache2 (2.4+)
Enable CGI-BIN and optionally even SSL if it is requiredAppend this at the end of /etc/apache2/apache2.conf# a2enmod cgi # a2enmod ssl
# vi /etc/apache2/apache2.conf AddHandler cgi-script .sh Alias /lpar2rrd "/home/lpar2rrd/lpar2rrd/www/" <Directory "/home/lpar2rrd/lpar2rrd/www/"> Options Indexes FollowSymLinks Require all granted </Directory> # CGI-BIN ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/" <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi"> AllowOverride None Options +ExecCGI +SymLinksIfOwnerMatch Require all granted SetHandler cgi-script </Directory> -
SuSE
Append at the end of /etc/apache2/httpd.conf following# vi /etc/apache2/httpd.conf AddHandler cgi-script .sh Alias /lpar2rrd "/home/lpar2rrd/lpar2rrd/www/" <Directory "/home/lpar2rrd/lpar2rrd/www/"> Options Indexes FollowSymLinks Require all granted </Directory> # CGI-BIN ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/" <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi"> AllowOverride None Options +ExecCGI +SymLinksIfOwnerMatch Require all granted SetHandler cgi-script </Directory> -
Apache 2.4+ generally
Configure httpd.conf, it might be in different locations (depends of distribution)- /etc/httpd/httpd.conf
- /etc/httpd/conf/httpd.conf
- /etc/opt/freeware/apache/httpd.conf (AIX)
- /opt/freeware/etc/httpd/httpd.conf (AIX)
- /opt/freeware/etc/httpd/conf/httpd.conf (AIX)
- /etc/apache2/apache2.conf
- Verify alias_module and cgi_module is loaded (uncommented it, the path can be different)
LoadModule alias_module /opt/freeware/lib/httpd/modules/mod_alias.so LoadModule cgi_module /opt/freeware/lib/httpd/modules/mod_cgi.so
- cgi_module could depend on mpm_prefork_module like here.
Allow it as well (uncomment it) and disable mpm_worker_module, mpm_event_moduleLoadModule mpm_prefork_module /opt/freeware/lib/httpd/modules/mod_mpm_prefork.so # LoadModule mpm_worker_module /opt/freeware/lib/httpd/modules/mod_mpm_worker.so # LoadModule mpm_event_module modules/mod_mpm_event.so <IfModule mpm_prefork_module> LoadModule cgi_module /opt/freeware/lib/httpd/modules/mod_cgi.so </IfModule> - Append this at the end of httpd.conf
Sometimes appears necessary to remove "Require all granted" lines in both directives.
AddHandler cgi-script .sh Alias /lpar2rrd "/home/lpar2rrd/lpar2rrd/www/" <Directory "/home/lpar2rrd/lpar2rrd/www/"> Options Indexes FollowSymLinks Require all granted </Directory> # CGI-BIN ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/" <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi"> AllowOverride None SetHandler cgi-script Options ExecCGI FollowSymLinks Require all granted </Directory>
-
Apache 1.3 - 2.3
Configure httpd.conf, it might be in different locations (depends of distribution)- /etc/httpd/httpd.conf
- /etc/httpd/conf/httpd.conf
# vi /etc/opt/freeware/apache/httpd.conf Alias /lpar2rrd "/home/lpar2rrd/lpar2rrd/www/" <Directory "/home/lpar2rrd/lpar2rrd/www/"> Options Indexes FollowSymLinks Order allow,deny Allow from all </Directory> # CGI-BIN ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/" <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi"> AllowOverride None Options ExecCGI FollowSymLinks Order allow,deny Allow from all </Directory> -
SELinux changes
You need to disable the SELinux service protection to be able to utilize basic web services.
You must do it in case you get "(13)Permission denied:" in Apache error_log.
You have 2 options:
- Disable SELinux only for Apache
Step 1. Query for the Boolean value you need to change: # getsebool -a | grep httpd_dis httpd_disable_trans --> off Step 2. Disable the SELinux protection: # setsebool -P httpd_disable_trans=1 Step 3. Verify that the Boolean has changed: # getsebool -a | grep httpd_dis httpd_disable_trans --> on
- Globally disable SELinux
# setenforce Permissive # vi /etc/sysconfig/selinux SELINUX=disabled
- Disable SELinux only for Apache
Skip it in case of configuring Virtual Appliance
Install LPAR2RRD server
-
Download the latest LPAR2RRD server
Upgrade your already running LPAR2RRD instance.
- Install it:
# su - lpar2rrd $ tar xvf lpar2rrd-7.XX.tar $ cd lpar2rrd-7.XX $ ./install.sh $ cd /home/lpar2rrd/lpar2rrd
- Schedule to run it from lpar2rrd crontab (it might already exist there)
Add if it does not exist as above$ crontab -l | grep load.sh $
$ crontab -e # LPAR2RRD UI 0,30 * * * * /home/lpar2rrd/lpar2rrd/load.sh > /home/lpar2rrd/lpar2rrd/load.out 2>&1
Assure there is just one such entry in crontab.
- You might need to add lpar2rrd user into /var/adm/cron/cron.allow (/etc/cron.allow on CentOS 8) if crontab command fails
Allow it for lpar2rrd user as root user.# echo "lpar2rrd" >> /var/adm/cron/cron.allow
- Assure you have a cron jobs for oVirt in place (upgrade script might do it automatically)
Skip it on the Virtual Appliance, it is already there.Add it if it does not exist like above$ crontab -l | grep "load_ovirt.sh" $
$ crontab -e # oVirt/RHV support 0,20,40 * * * * /home/lpar2rrd/lpar2rrd/load_ovirt.sh > /home/lpar2rrd/lpar2rrd/load_ovirt.out 2>&1
-
Initial start from cmd line:
$ cd /home/lpar2rrd/lpar2rrd $ ./load.sh
- Go to the web UI: http://<your web server>/lpar2rrd/
Use Ctrl-F5 to refresh the web browser cache.
Troubleshooting
-
If you have any problems with the UI then check:
(note that the path to Apache logs might be different, search apache logs in /var)tail /var/log/httpd/error_log # Apache error log tail /var/log/httpd/access_log # Apache access log tail /var/tmp/lpar2rrd-realt-error.log # STOR2RRD CGI-BIN log tail /var/tmp/systemd-private*/tmp/lpar2rrd-realt-error.log # STOR2RRD CGI-BIN log when Linux has enabled private temp
- Test of CGI-BIN setup
umask 022 cd /home/lpar2rrd/lpar2rrd/ cp bin/test-healthcheck-cgi.sh lpar2rrd-cgi/
go to the web browser: http://<your web server>/lpar2rrd/test.html
You should see your Apache, LPAR2RRD, and Operating System variables, if not, then check Apache logs for connected errors
OS agent is add-on feature for monitoring from operating system level.
It is monitoring CPU, memory utilization, paging, LAN and SAN traffic on all adapters.
It requires the OS agent deployment to every monitored VM or KVM host.
The agent is written in Perl and calls basic OS commands to obtain required statistics like vmstat, iostat.
 |
Additional information about the OS agent:
- OS agent description
- OS agent release notes and download
- Features supported by the OS agent.
Prerequisites
- Perl
- Opened TCP communication between each VM and LPAR2RRD server on port 8162.
Connections are initiated from VM side. - Additional disk space on LPAR2RRD server (about 40MB per each monitored VM)
-
Create preferable dedicated user lpar2rrd on each VM with minimum rights
# useradd -c "LPAR2RRD agent user" -m lpar2rrd
OS agent installation (client)
-
Get the latest OS agent from download page
- Linux installation under root
# rpm -Uvh lpar2rrd-agent-6.00-0.noarch.rpm # rpm -qa|grep lpar2rrd-agent lpar2rrd-agent-6.00-0
- Linux Debian
# apt-get install lpar2rrd-agent_6.00-0_all.deb lpar2rrd-agent-6.00-0
- Solaris x86 installation under root:
# gunzip lpar2rrd-agent-6.00-0.solaris-i86pc.tar.gz # tar xf lpar2rrd-agent-6.00-0.solaris-i86pc.tar # pkgadd -d . The following packages are available: 1 lpar2rrd-agent LPAR2RRD OS agent 6.00 (i86pc) 6.00 ...Solaris upgrade under root:# pkgrm lpar2rrd-agent # pkgadd -d .
- Schedule its run every minute from the crontab on every VM.
This line must be placed into lpar2rrd crontab:# su - lpar2rrd $ crontab -e * * * * * /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl <LPAR2RRD-SERVER> > /var/tmp/lpar2rrd-agent.out 2>&1
Replace <LPAR2RRD-SERVER> by hostname of your LPAR2RRD server.
-
You might need to add lpar2rrd user into /var/adm/cron/cron.allow under root user if above "crontab -e" fails.
# echo "lpar2rrd" >> /var/adm/cron/cron.allow
LPAR2RRD server (daemon)
-
Edit etc/lpar2rrd.cfg and set following (if it is not already set):
$ vi /home/lpar2rrd/lpar2rrd/etc/lpar2rrd.cfg LPAR2RRD_AGENT_DAEMON=1
-
The daemon is started when load.sh starts
$ ./load.sh Starting LPAR2RRD daemon on port:8162 ...
- Assure it is running and listening on port 8162:
$ ps -ef|grep lpar2rrd-daemon lpar2rrd 10617010 1 0 Mar 16 - 0:00 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/lpar2rrd-daemon.pl $ netstat -an| grep 8162 tcp4 0 0 *.8162 *.* LISTEN
- OS agent data graphs will appear in the UI, use Ctrl-F5 to refresh your web browser
Troubleshooting
-
Client (agent) side:
-
Test if communication through the LAN is allowed.
$ telnet <LPAR2RRD-SERVER> 8162 Connected to 192.168.1.1 . Escape character is '^]'.
This is ok, exit either Ctrl-C or ^].
-
Check following agent files:
data store: /var/tmp/lpar2rrd-agent-*.txt
error log: /var/tmp/lpar2rrd-agent-*.err
output log: /var/tmp/lpar2rrd-agent.out
-
run the agent from cmd line:
$ /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <LPAR2RRD-SERVER> ... Agent send : yes : forced by -d Agent send slp: sending wait: 4 OS agent working for server: <LPAR2RRD-SERVER> store file for sending is /var/tmp/lpar2rrd-agent-<LPAR2RRD-SERVER>-lpar2rrd.txt
It means that data has been sent to the server, all is fine
Here is example when the agent is not able to sent data :$ /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <LPAR2RRD-SERVER> ... Agent send : yes : forced by -d Agent send slp: sending wait: 1 OS agent working for server: <LPAR2RRD-SERVER> store file for sending is /var/tmp/lpar2rrd-agent-<LPAR2RRD-SERVER>-lpar2rrd.txt Agent timed out after : 50 seconds /opt/lpar2rrd-agent/lpar2rrd-agent.pl:265
It means that the agent could not contact the server.
Check communication, port, above telnet example, DNS resolution of the server etc.
-
Test if communication through the LAN is allowed.
-
Server side:
- test if the daemon on LPAR2RRD server is running, and checking the logs
$ ps -ef|grep lpar2rrd-daemon lpar2rrd 10617010 1 0 Mar 16 - 0:00 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/lpar2rrd-daemon.pl $ cd /home/lpar2rrd/lpar2rrd $ tail logs/error.log-daemon $ tail logs/daemon.out new server has been found and registered: Linux (lpar=linuxhost01) mkdir : /lpar2rrd/data/Linux/no_hmc/linuxhost01/
It means that new OS agent has been registered from linuxhost01 (Linux stand-alone example)
-
Test if OS agent data is being stored on the LPAR2RRD server and have actual timestamp:
$ cd /home/lpar2rrd/lpar2rrd $ ls -l data/<server name>/*/<VM name>/*mmm -rw-r--r-- 2 lpar2rrd staff 7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/cpu.mmm -rw-r--r-- 2 lpar2rrd staff 7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/lan-en1.mmm -rw-r--r-- 2 lpar2rrd staff 10790264 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/mem.mmm -rw-r--r-- 2 lpar2rrd staff 7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/pgs.mmm -rw-r--r-- 2 lpar2rrd staff 7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/san-vscsi0.mmm -rw-r--r-- 2 lpar2rrd staff 3597208 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/san_resp-vscsi0.mmm $ find data -name mem.mmm -exec ls -l {} \; ...
- test if the daemon on LPAR2RRD server is running, and checking the logs
-
In case of a problem check our forum or contact us via support@lpar2rrd.com.
We would need this data for start of troubleshooting.
Notes
-
You will not need to upgrade LPAR2RRD agents regularly with each LPAR2RRD upgrade.
Read release notes if that is necessary.
Check OS agent upgrade steps.
Due to a bug in LPAR2RRD v7.50 you will not be able to add new monitored oVirt/RHV instance
Upgrade to LPAR2RRD v7.51-1 to fix it
Implementation is agent less.
Data source is ovirt_engine_history database which is part of Data Warehouse on oVirt/RHV Virtualization Manager.
Tool accesses the ovirt_engine_history database remotely via its SQL API.
ovirt_engine_history database contains only historical performance data. There is only the admin account which we use.
Due to API changes in RHV 4.5 use LPAR2RRD v 7.42-13 or newer, olders do not work
Upgrade to LPAR2RRD v7.51-1 to fix it
 |
Implementation is agent less.
Data source is ovirt_engine_history database which is part of Data Warehouse on oVirt/RHV Virtualization Manager.
Tool accesses the ovirt_engine_history database remotely via its SQL API.
ovirt_engine_history database contains only historical performance data. There is only the admin account which we use.
Due to API changes in RHV 4.5 use LPAR2RRD v 7.42-13 or newer, olders do not work
OS Prerequisites
-
Open network connection to the RedHat Virtualization Manager on port 5432
$ perl /home/lpar2rrd/lpar2rrd/bin/conntest.pl <Virtualization Manager/IP> 5432 Connection to "<Virtualization Manager hostname/IP>" on port "80" is ok
-
Make sure you have installed perl modules: perl-DBI and perl-DBD-Pg on LPAR2RRD server
Install them if they are missing under root user
$ rpm -q perl-DBI perl-DBD-Pg perl-DBI-1.627-4.el7.x86_64 perl-DBD-Pg-2.19.3-4.el7.x86_64
- Linux RedHat, CentOS
# yum install perl-DBD-Pg
- Linux Debian, Ubuntu
# apt-get install libdbd-pg-perl
-
AIX
- yum installation
Download perl-DBI-1.623-1 and perl-DBD-Pg-3.7.4
# yum install postgresql-libs # rpm -i perl-DBI-1.623-1.aix5.1.ppc.rpm # rpm -i perl-DBD-Pg-3.7-4.ppc.rpm
- manual rpm installation
- yum installation
- Linux RedHat, CentOS
Database access
-
Each Virtualization Manager has its own randomly generated password for ovirt_engine_history database.
To get the database password, you have to connect to your RHV / oVirt engine via SSH and run this command:
# grep DWH_DB_PASSWORD /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-database.conf
# grep DWH_DB_USER /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-database.conf ovirt_engine_history
Attach oVirt / RHV
-
All is configured from the UI.
- Assure you have a cron jobs for oVirt in place (upgrade script might do it automatically)
Skip it on the Virtual Appliance, it is already there.Add it if it does not exist like above$ crontab -l | grep "load_ovirt.sh" $
$ crontab -e # oVirt/RHV support 0,20,40 * * * * /home/lpar2rrd/lpar2rrd/load_ovirt.sh > /home/lpar2rrd/lpar2rrd/load_ovirt.out 2>&1
- Wait 30 minutes and then go to the web UI: http://<your web server>/lpar2rrd/
Use Ctrl-F5 to refresh the web browser cache.