Kubernetes Monitoring

Skip Prerequisites, Web and LPAR2RRD tabs in case of configuring Virtual Appliance, Docker or a Container

HW sizing


Consider usage of our brand new full-stack infrastructure monitoring tool XorMon Next Generation as LPAR2RRD replacement.
It brings a new level of infrastructure monitoring by relying on a modern technology stack.
In particular, reporting, exporting, alerting and presentation capabilities are unique on the market.

Follow installation procedure for your operating system platform
Implementation is agentless, all data is gathered from Kubernetes API & Kubernetes Metrics server API

Kubernetes performance monitoring schema

Check if you have metrics-server installed

    kubectl get pods -n kube-system
    Kubernetes performance monitoring metrics server

Create a lpar2rrd serviceaccount

    $ kubectl create serviceaccount lpar2rrd

Create a clusterrole

    Create file ClusterRole.yml with this content: 
    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: lpar2rrd
rules:
- apiGroups: ["","metrics.k8s.io"]
  resources: ["pods","nodes","services","nodes/proxy", "endpoints", "namespaces"]
  verbs: ["get", "watch", "list"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
    Apply it by: 
    $ kubectl apply -f ClusterRole.yml

Create a clusterrolebinding

    Create file ClusterRoleBinding.yml with this content: 
    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: lpar2rrd
subjects:
- kind: ServiceAccount
  name: lpar2rrd
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: lpar2rrd
    Make sure it is the same namespace as the serviceaccount like default above.

    Apply it by: 
    $ kubectl apply -f ClusterRoleBinding.yml

Get lpar2rrd token

    Create file Lpar2rrdSecret.yml with this content: 
    apiVersion: v1
kind: Secret
metadata:
  name: lpar2rrd-token
  annotations:
    kubernetes.io/service-account.name: lpar2rrd
type: kubernetes.io/service-account-token
    Apply it by: 
    $ kubectl apply -f Lpar2rrdSecret.yml
    Get lpar2rrd token: 
    $ kubectl describe secret lpar2rrd-token
    Kubernetes performance monitoring

Get cluster endpoint

    $ kubectl config view
    Kubernetes performance monitoring

Configure Kubernetes in LPAR2RRD

  • LPAR2RRD UI ➡ Settings icon ➡ Kubernetes ➡ New
    Storage management

  • Make sure cron job for Kubernetes is in place (upgrade script might do it for you automatically)
    Skip this step if you install the Virtual Appliance - it is already taken care of. 
    $ crontab -l | grep "load_kubernetes.sh"
$
    Add following lines to crontab if necessary 
    $ crontab -e

# Kubernetes support
0,20,40 * * * *  /home/lpar2rrd/lpar2rrd/load_kubernetes.sh > /home/lpar2rrd/lpar2rrd/load_kubernetes.out 2>&1
  • Wait 30 minutes and then go to the web UI: http://<your web server>/lpar2rrd/
    Use Ctrl-F5 to refresh the web browser cache.


Install LPAR2RRD server (all under lpar2rrd user)

  • Download the latest LPAR2RRD server
    Upgrade your already running LPAR2RRD instance.

  • Install it: 
    # su - lpar2rrd
$ tar xvf lpar2rrd-7.XX.tar
$ cd lpar2rrd-7.XX
$ ./install.sh
$ cd /home/lpar2rrd/lpar2rrd
  • Make sure all Perl modules are in place
    cd /home/lpar2rrd/lpar2rrd
. etc/lpar2rrd.cfg; $PERL bin/perl_modules_check.pl
    If there is missing "LWP::Protocol::https" then check this docu to fix it

  • Enable Apache authorisation 
    umask 022
cd /home/lpar2rrd/lpar2rrd
cp html/.htaccess www
cp html/.htaccess lpar2rrd-cgi
  • Schedule to run it from lpar2rrd crontab (it might already exist there)
    $ crontab -l | grep load.sh
$
    Add if it does not exist as above 
    $ crontab -e

# LPAR2RRD UI
0,30 * * * * /home/lpar2rrd/lpar2rrd/load.sh > /home/lpar2rrd/lpar2rrd/load.out 2>&1
    Assure there is just one such entry in crontab.

  • You might need to add lpar2rrd user into /etc/cron.allow (Linux) or /var/adm/cron/cron.allow (AIX) if 'crontab -e' command fails
    Allow it for lpar2rrd user as root user. 
    # echo "lpar2rrd" >> /etc/cron.allow
  • Assure you have a cron job for Kubernetes is in place (upgrade script might do it automatically) Skip it on the Virtual Appliance, it is already there. 
    $ crontab -l | grep "load_kubernetes.sh"
$
    Add it if it does not exist like above 
    $ crontab -e

# Kubernetes support
0,20,40 * * * *  /home/lpar2rrd/lpar2rrd/load_kubernetes.sh > /home/lpar2rrd/lpar2rrd/load_kubernetes.out 2>&1
  • Initial start from cmd line: 
    $ cd /home/lpar2rrd/lpar2rrd
$ ./load.sh
  • Go to the web UI: http://<your web server>/lpar2rrd/
    Use Ctrl-F5 to refresh the web browser cache.

Troubleshooting

  • If you have any problems with the UI then check:
    (note that the path to Apache logs might be different, search apache logs in /var) 
    tail /var/log/httpd/error_log             # Apache error log
tail /var/log/httpd/access_log            # Apache access log
tail /var/tmp/lpar2rrd-realt-error.log    # STOR2RRD CGI-BIN log
tail /var/tmp/systemd-private*/tmp/lpar2rrd-realt-error.log # STOR2RRD CGI-BIN log when Linux has enabled private temp
  • Test of CGI-BIN setup 
    umask 022
cd /home/lpar2rrd/lpar2rrd/
cp bin/test-healthcheck-cgi.sh lpar2rrd-cgi/
    go to the web browser: http://<your web server>/lpar2rrd/test.html
    You should see your Apache, LPAR2RRD, and Operating System variables, if not, then check Apache logs for connected errors