Kubernetes Monitoring

Skip Prerequisites, Web and LPAR2RRD tabs in case of configuring Virtual Appliance, Docker or a Container

Follow installation procedure for your operating system platform
Implementation is agentless, all data is gathered from Kubernetes API & Kubernetes Metrics server API

Kubernetes performance monitoring schema

Check if you have metrics-server installed

    kubectl get pods -n kube-system
    Kubernetes performance monitoring metrics server

Create a lpar2rrd serviceaccount

    $ kubectl create serviceaccount lpar2rrd

Create a clusterrole

    Create file ClusterRole.yml with this content: 
    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: lpar2rrd
rules:
- apiGroups: ["","metrics.k8s.io"]
  resources: ["pods","nodes","services","nodes/proxy", "endpoints", "namespaces"]
  verbs: ["get", "watch", "list"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
    Apply it by: 
    $ kubectl apply -f ClusterRole.yml

Create a clusterrolebinding

    Create file ClusterRoleBinding.yml with this content: 
    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: lpar2rrd
subjects:
- kind: ServiceAccount
  name: lpar2rrd
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: lpar2rrd
    Make sure it is the same namespace as the serviceaccount like default above.

    Apply it by: 
    $ kubectl apply -f ClusterRoleBinding.yml

Get lpar2rrd token

    Create file Lpar2rrdSecret.yml with this content: 
    apiVersion: v1
kind: Secret
metadata:
  name: lpar2rrd-token
  annotations:
    kubernetes.io/service-account.name: lpar2rrd
type: kubernetes.io/service-account-token
    Apply it by: 
    $ kubectl apply -f Lpar2rrdSecret.yml
    Get lpar2rrd token: 
    $ kubectl describe secret lpar2rrd-token
    Kubernetes performance monitoring

Get cluster endpoint

    $ kubectl config view
    Kubernetes performance monitoring

Configure Kubernetes in LPAR2RRD

  • LPAR2RRD UI ➡ Settings icon ➡ Kubernetes ➡ New
    Storage management

  • Make sure cron job for Kubernetes is in place (upgrade script might do it for you automatically)
    Skip this step if you install the Virtual Appliance - it is already taken care of. 
    $ crontab -l | grep "load_kubernetes.sh"
$
    Add following lines to crontab if necessary 
    $ crontab -e

# Kubernetes support
0,20,40 * * * *  /home/lpar2rrd/lpar2rrd/load_kubernetes.sh > /home/lpar2rrd/lpar2rrd/load_kubernetes.out 2>&1
  • Wait 30 minutes and then go to the web UI: http://<your web server>/lpar2rrd/
    Use Ctrl-F5 to refresh the web browser cache.


Install LPAR2RRD server

  • Download the latest LPAR2RRD server
    Upgrade your already running LPAR2RRD instance.

  • Install it: 
    # su - lpar2rrd
$ tar xvf lpar2rrd-7.XX.tar
$ cd lpar2rrd-7.XX
$ ./install.sh
$ cd /home/lpar2rrd/lpar2rrd
  • Make sure all Perl modules are in place
    cd /home/lpar2rrd/lpar2rrd
. etc/lpar2rrd.cfg; $PERL bin/perl_modules_check.pl
    If there is missing "LWP::Protocol::https" then check this docu to fix it

  • Enable Apache authorisation 
    umask 022
cd /home/lpar2rrd/lpar2rrd
cp html/.htaccess www
cp html/.htaccess lpar2rrd-cgi
  • Schedule to run it from lpar2rrd crontab (it might already exist there)
    $ crontab -l | grep load.sh
$
    Add if it does not exist as above 
    $ crontab -e

# LPAR2RRD UI
0,30 * * * * /home/lpar2rrd/lpar2rrd/load.sh > /home/lpar2rrd/lpar2rrd/load.out 2>&1
    Assure there is just one such entry in crontab.

  • You might need to add lpar2rrd user into /var/adm/cron/cron.allow (/etc/cron.allow on CentOS 8) if crontab command fails
    Allow it for lpar2rrd user as root user. 
    # echo "lpar2rrd" >> /var/adm/cron/cron.allow
  • Assure you have a cron job for Kubernetes is in place (upgrade script might do it automatically) Skip it on the Virtual Appliance, it is already there. 
    $ crontab -l | grep "load_kubernetes.sh"
$
    Add it if it does not exist like above 
    $ crontab -e

# Kubernetes support
0,20,40 * * * *  /home/lpar2rrd/lpar2rrd/load_kubernetes.sh > /home/lpar2rrd/lpar2rrd/load_kubernetes.out 2>&1
  • Initial start from cmd line: 
    $ cd /home/lpar2rrd/lpar2rrd
$ ./load.sh
  • Go to the web UI: http://<your web server>/lpar2rrd/
    Use Ctrl-F5 to refresh the web browser cache.

Troubleshooting

  • If you have any problems with the UI then check:
    (note that the path to Apache logs might be different, search apache logs in /var) 
    tail /var/log/httpd/error_log             # Apache error log
tail /var/log/httpd/access_log            # Apache access log
tail /var/tmp/lpar2rrd-realt-error.log    # STOR2RRD CGI-BIN log
tail /var/tmp/systemd-private*/tmp/lpar2rrd-realt-error.log # STOR2RRD CGI-BIN log when Linux has enabled private temp
  • Test of CGI-BIN setup 
    umask 022
cd /home/lpar2rrd/lpar2rrd/
cp bin/test-healthcheck-cgi.sh lpar2rrd-cgi/
    go to the web browser: http://<your web server>/lpar2rrd/test.html
    You should see your Apache, LPAR2RRD, and Operating System variables, if not, then check Apache logs for connected errors