ssh based checks

Our scenario involves checking /var/log not exceeding a given file system size. A violation may indicate e.g. logrotate not being set up properly. The subsequently described steps assume the Icinga software running on host sdi10a.mi.hdm-stuttgart.de / 141.62.75.120 querying target host sdi10b.mi.hdm-stuttgart.de.

Target host configuration steps:

  1. Download the check_file_size.sh plugin to /usr/lib/nagios/plugins.

  2. Test check_file_size.sh manually by choosing appropriate warning and critical threshold values:

    sdi10b#> /usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn 1000 --maxcrit 1500 
    FILE Critical: Size of 1610 > 1500 for /var/log
    
    sdi10b#> /usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn 1000 --maxcrit 2000 
    FILE Warning: Size of 1610 > 1000 for /var/log
    
    sdi10b#> /usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn 2000 --maxcrit 4000 
    FILE OK: All files (1) fall within requested parameters
    
  3. Prepare for remote execution. Create a directory /etc/nagiosBySsh and the following executable bash script /etc/nagiosBySsh/nagioscheckssh within:

    #!/bin/bash
    
    if [ -z "$SSH_ORIGINAL_COMMAND" ]; then
        echo "Environment variable «SSH_ORIGINAL_COMMAND» undefined"
    else
        exec $SSH_ORIGINAL_COMMAND
    fi
    
    #end

    This script will later be executed by a remote ssh call. The actual command to be executed will be provided by the environment variable SSH_ORIGINAL_COMMAND.

  4. Simulate remote execution by using env for setting the desired environment variable:

    sdi10b#> env SSH_ORIGINAL_COMMAND='/usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn 1000 --maxcrit 1500' /etc/nagiosBySsh/nagioscheckssh
    

    Depending on the chosen parameter values the resulting output should be similar to:

    FILE Critical: Size of 1610 > 1500 for /var/log
  5. For the sake of security a non-privileged account will be used for remote ssh execution. Create a system account nagioscheck for this purpose:

    adduser --shell /bin/bash --system nagioscheck
  6. As user nagioscheck create an ssh pair of keys using an empty pass phrase:

    sdi10b#> su - nagioscheck
    sdi10b$> ssh-keygen 
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/nagioscheck/.ssh/id_rsa): 
    Created directory '/home/nagioscheck/.ssh'.
    Enter passphrase (empty for no passphrase): 
    Enter same passphrase again: 
    Your identification has been saved in /home/nagioscheck/.ssh/id_rsa. ❶
    Your public key has been saved in /home/nagioscheck/.ssh/id_rsa.pub. ❷
    The key fingerprint is: ...
  7. The private ssh key ❶ will later be used for remote query invocations. After moving it to the querying host it may deleted on the target host for security reasons.

    Configure the corresponding public key ❷ to allow for ordinary remote ssh login:

    sdi10b#> su - nagioscheck
    sdi10b> cd ~/.ssh
    sdi10b> cp id_rsa.pub authorized_keys
  8. We now restrict remote logins to execution of /etc/nagiosBySsh/nagioscheckssh from 3. Modify /home/nagioscheck/.ssh/authorized_keys to contain:

    from="141.62.75.110",command="/etc/nagiosBySsh/nagioscheckssh" ssh-rsa AAAAB3N...LKPFSJo5 nagioscheck@sdi10b

    Explanation: ssh connections origination from 141.62.75.110 will receive the execution result of /etc/nagiosBySsh/nagioscheckssh. In particular no login shell will be provided.

    Caution

    The from option really requires an IP rather than a DNS name.

Icinga host configuration steps:

  1. Copy the target host's /home/nagioscheck/.ssh/id_rsa private key to /etc/icinga2/nagioscheck_id_rsa. Since the Icinga process running with the nagios user is requires read access we change the ownnership accordingly:

    chown nagios.nagios nagioscheck_id_rsa
  2. The Icinga daemon runs with the effective user's id nagios.We thus have to assure remote ssh login as user nagios:

    ssh -i /etc/icinga2/nagioscheck_id_rsa nagioscheck@sdi10b.mi.hdm-stuttgart.de \
      "/usr/lib/nagios/plugins/check_file_size.sh /var/log \
      --maxwarn 1000 --maxcrit 1500"
    The authenticity of host 'sdi10b.mi.hdm-stuttgart.de (141.62.75.120)' can't be established.
    ECDSA key fingerprint is SHA256:4L5rfTIJPu1lr1gpTyvaywDE01W55roZjNGKkni/060.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'sdi10b.mi.hdm-stuttgart.de,141.62.75.120' (ECDSA) to the list of known hosts.
    du: cannot read directory '/var/log/samba': Permission denied
    du: cannot read directory '/var/log/unattended-upgrades': Permission denied
    FILE Critical: Size of 1742 > 1500 for /var/log

    Explanation: Remote command execution of /usr/lib/nagios/plugins/check_file_size.sh as user nagioscheck lacks privileges to completely read the entire /var/log directory tree. This will be fixed in the next step.

  3. Turn back to your target server and allow sudo execution of /usr/lib/nagios/plugins/check_file_size.sh to user nagioscheck without providing a password. Start by:

    sdi10b#> apt-get install sudo

    Subsequently create a file /etc/sudoers.d/nagios to contain:

    nagioscheck     ALL=NOPASSWD : /usr/lib/nagios/plugins/check_file_size.sh

    You should now be able to initiate sudo privileged execution from your Icinga host:

    ssh -i /etc/icinga2/nagioscheck_id_rsa nagioscheck@sdi10b.mi.hdm-stuttgart.de \
      "sudo /usr/lib/nagios/plugins/check_file_size.sh /var/log \
      --maxwarn 1000 --maxcrit 1500"
    FILE Critical: Size of 1726 > 1500 for /var/log
  4. Configure an Icinga check command in conf.d/commands.conf:

    object CheckCommand "by_ssh_file_size" {
      import "by_ssh"
    
      vars.by_ssh_command  = "sudo /usr/lib/nagios/plugins/check_file_size.sh --maxwarn $by_ssh_file_size_warn$ --maxcrit $by_ssh_file_size_crit$ $by_ssh_file_size_path$"
      vars.by_ssh_identity = "/etc/icinga2/nagioscheck_id_rsa"
      vars.by_ssh_logname  = "nagioscheck"
    
    #  Parameters by_ssh_file_size_warn, vars.by_ssh_file_size_crit and
    #  by_ssh_file_size_path will be defined in service or host definition
    }
  5. Define an Icinga template in conf.d/services.conf:

    apply Service for (path => config in host.vars.paths) {
      import "generic-service"
    
      check_command = "by_ssh_file_size"
    
      vars += config
    }
  6. Finally add two suitable host service checks in conf.d/hosts.conf:

    object Host "sdi10b.mi.hdm-stuttgart.de" {
      import "generic-host"
    
      address = "sdi10b.mi.hdm-stuttgart.de"
    
      vars.paths["/var/log size"] = {
        by_ssh_file_size_warn = "500"
        by_ssh_file_size_crit = "5500"
        by_ssh_file_size_path = "/var/log"
      }
    
      vars.paths["/var/lib size"] = {
        by_ssh_file_size_warn = "50000"
        by_ssh_file_size_crit = "900000"
        by_ssh_file_size_path = "/var/lib"
      }
    
      vars.notification["mail"] = {
        groups = [ "icingaadmins" ]
      }
    }