Kubernetes Monitoring

Skip Prerequisites, Web and LPAR2RRD tabs in case of configuring Virtual Appliance, Docker or a Container

Follow installation procedure for your operating system platform
Implementation is agentless, all data is gathered from Kubernetes API & Kubernetes Metrics server API

Kubernetes performance monitoring schema

Check if you have metrics-server installed

    kubectl get pods -n kube-system
    
    Kubernetes performance monitoring metrics server

Create a lpar2rrd serviceaccount

    $ kubectl create serviceaccount lpar2rrd
    

Create a clusterrole

    Create file ClusterRole.yml with this content:
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: lpar2rrd
    rules:
    - apiGroups: ["","metrics.k8s.io"]
      resources: ["pods","nodes","services","nodes/proxy", "endpoints", "namespaces"]
      verbs: ["get", "watch", "list"]
    - nonResourceURLs: ["/metrics"]
      verbs: ["get"]
    
    Apply it by:
    $ kubectl apply -f ClusterRole.yml
    

Create a clusterrolebinding

    Create file ClusterRoleBinding.yml with this content:
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: lpar2rrd
    subjects:
    - kind: ServiceAccount
      name: lpar2rrd
      namespace: default
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: lpar2rrd
    
    Make sure it is the same namespace as the serviceaccount like default above.

    Apply it by:
    $ kubectl apply -f ClusterRoleBinding.yml
    

Get lpar2rrd token

    Create file Lpar2rrdSecret.yml with this content:
    apiVersion: v1
    kind: Secret
    metadata:
      name: lpar2rrd-token
      annotations:
        kubernetes.io/service-account.name: lpar2rrd
    type: kubernetes.io/service-account-token
    
    Apply it by:
    $ kubectl apply -f Lpar2rrdSecret.yml 
    
    Get lpar2rrd token:
    $ kubectl describe secret lpar2rrd-token 
    
    Kubernetes performance monitoring

Get cluster endpoint

    $ kubectl config view
    
    Kubernetes performance monitoring

Configure Kubernetes in LPAR2RRD

  • LPAR2RRD UI ➡ Settings icon ➡ Kubernetes ➡ New
    Storage management

  • Make sure cron job for Kubernetes is in place (upgrade script might do it for you automatically)
    Skip this step if you install the Virtual Appliance - it is already taken care of.
    $ crontab -l | grep "load_kubernetes.sh"
    $
    
    Add following lines to crontab if necessary
    $ crontab -e
    
    # Kubernetes support
    0,20,40 * * * *  /home/lpar2rrd/lpar2rrd/load_kubernetes.sh > /home/lpar2rrd/lpar2rrd/load_kubernetes.out 2>&1
    
  • Wait 30 minutes and then go to the web UI: http://<your web server>/lpar2rrd/
    Use Ctrl-F5 to refresh the web browser cache.


Install LPAR2RRD server

  • Download the latest LPAR2RRD server
    Upgrade your already running LPAR2RRD instance.

  • Install it:
    # su - lpar2rrd
    $ tar xvf lpar2rrd-7.XX.tar
    $ cd lpar2rrd-7.XX
    $ ./install.sh
    $ cd /home/lpar2rrd/lpar2rrd
    
  • Make sure all Perl modules are in place
    cd /home/lpar2rrd/lpar2rrd
    . etc/lpar2rrd.cfg; $PERL bin/perl_modules_check.pl
    
    If there is missing "LWP::Protocol::https" then check this docu to fix it

  • Enable Apache authorisation
    umask 022
    cd /home/lpar2rrd/lpar2rrd
    cp html/.htaccess www
    cp html/.htaccess lpar2rrd-cgi
    
  • Schedule to run it from lpar2rrd crontab (it might already exist there)
    $ crontab -l | grep load.sh
    $
    
    Add if it does not exist as above
    $ crontab -e
    
    # LPAR2RRD UI
    0,30 * * * * /home/lpar2rrd/lpar2rrd/load.sh > /home/lpar2rrd/lpar2rrd/load.out 2>&1 
    
    Assure there is just one such entry in crontab.

  • You might need to add lpar2rrd user into /var/adm/cron/cron.allow (/etc/cron.allow on CentOS 8) if crontab command fails
    Allow it for lpar2rrd user as root user.
    # echo "lpar2rrd" >> /var/adm/cron/cron.allow
    
  • Assure you have a cron job for Kubernetes is in place (upgrade script might do it automatically) Skip it on the Virtual Appliance, it is already there.
    $ crontab -l | grep "load_kubernetes.sh"
    $
    
    Add it if it does not exist like above
    $ crontab -e
    
    # Kubernetes support
    0,20,40 * * * *  /home/lpar2rrd/lpar2rrd/load_kubernetes.sh > /home/lpar2rrd/lpar2rrd/load_kubernetes.out 2>&1 
    
  • Initial start from cmd line:
    $ cd /home/lpar2rrd/lpar2rrd
    $ ./load.sh
    
  • Go to the web UI: http://<your web server>/lpar2rrd/
    Use Ctrl-F5 to refresh the web browser cache.

Troubleshooting

  • If you have any problems with the UI then check:
    (note that the path to Apache logs might be different, search apache logs in /var)
    tail /var/log/httpd/error_log             # Apache error log
    tail /var/log/httpd/access_log            # Apache access log
    tail /var/tmp/lpar2rrd-realt-error.log    # STOR2RRD CGI-BIN log
    tail /var/tmp/systemd-private*/tmp/lpar2rrd-realt-error.log # STOR2RRD CGI-BIN log when Linux has enabled private temp
    
  • Test of CGI-BIN setup
    umask 022
    cd /home/lpar2rrd/lpar2rrd/
    cp bin/test-healthcheck-cgi.sh lpar2rrd-cgi/
    
    go to the web browser: http://<your web server>/lpar2rrd/test.html
    You should see your Apache, LPAR2RRD, and Operating System variables, if not, then check Apache logs for connected errors