<?xml version="1.0" encoding="UTF-8"?>
<chapter version="5.1" xml:id="sdi_icinga2"
         xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xila="http://www.w3.org/2001/XInclude/local-attributes"
         xmlns:xi="http://www.w3.org/2001/XInclude"
         xmlns:trans="http://docbook.org/ns/transclusion"
         xmlns:svg="http://www.w3.org/2000/svg"
         xmlns:m="http://www.w3.org/1998/Math/MathML"
         xmlns:html="http://www.w3.org/1999/xhtml"
         xmlns:db="http://docbook.org/ns/docbook">
  <title>Icinga</title>

  <para>With respect to more current modules we start from the more recent
  <link xlink:href="https://packages.icinga.com/debian/#indexlist">icinga2
  packages</link>.</para>

  <itemizedlist>
    <listitem>
      <para><link
      xlink:href="https://www.howtoforge.com/tutorial/how-to-install-icinga2-on-debian-9">How
      to Install Icinga 2 Monitoring Tool on Debian 9.2</link></para>

      <caution>
        <para>Fix <code>admin</code> email in
        <filename>/etc/icinga2/conf.d/users.conf</filename> prior to starting
        **any** <productname>Icinga</productname> related service:</para>

        <informaltable border="0">
          <tr>
            <td valign="top"><programlisting language="none">object User "icingaadmin" {
  import "generic-user"

  display_name = "Icinga 2 Admin"
  groups = [ "icingaadmins" ]

  email = "<emphasis role="red">root@localhost</emphasis>"
}</programlisting></td>

            <td valign="top"><programlisting language="none">object User "icingaadmin" {
  import "generic-user"

  display_name = "Icinga 2 Admin"
  groups = [ "icingaadmins" ]

  email = "<emphasis role="red">yourEmail@whatsoever.org</emphasis>"
}</programlisting></td>
          </tr>
        </informaltable>
      </caution>
    </listitem>

    <listitem>
      <para><link
      xlink:href="https://www.digitalocean.com/community/tutorials/how-to-install-icinga-and-icinga-web-on-ubuntu-16-04#step-2-%E2%80%93-installing-the-icinga-web-interface">Timezone
      considerations</link>.</para>
    </listitem>

    <listitem>
      <para><link xlink:href="https://icinga.com/docs/icinga2/latest">Icinga
      latest documentation.</link></para>
    </listitem>
  </itemizedlist>

  <para>Installation road map:</para>

  <orderedlist>
    <listitem>
      <para>Base system + <productname>Icinga</productname> web</para>
    </listitem>

    <listitem>
      <para>Optional: Setting up <productname>Icinga 2</productname> API
      (prerequisite for the <quote>director</quote> module)</para>
    </listitem>

    <listitem>
      <para>Configure <quote>director</quote> module.</para>
    </listitem>
  </orderedlist>

  <tip>
    <itemizedlist>
      <listitem>
        <para>Creating an initial <productname>Apache</productname> /
        <productname>Nginx</productname> configuration can be achieved
        using:</para>

        <screen>icingacli setup config webserver {<emphasis role="red">apache</emphasis>|<emphasis
            role="red">nginx</emphasis>} --document-root /usr/share/icingaweb2/public</screen>
      </listitem>

      <listitem>
        <para>Configuration validation:</para>

        <screen><command>icinga2</command> <option>daemon</option> -<option>-validate</option></screen>
      </listitem>

      <listitem>
        <para>Module <quote>setup</quote> activation being prerequisite for
        executing <command>icingacli</command> <option>setup</option>
        <option>...</option> commands:</para>

        <screen><command>icingacli</command> <option>module</option> <option>enable</option> <option>setup</option></screen>
      </listitem>

      <listitem>
        <para>Command feature activation:</para>

        <screen><command>icinga2</command> <option>feature</option> <option>enable</option> <option>command</option>
<command>systemctl</command> <option>restart</option> <option>icinga2.service</option></screen>
      </listitem>

      <listitem>
        <para>vim <productname>Icinga</productname> configuration file syntax
        highlighting:</para>

        <screen><command>aptitude</command> <option>install</option> <option>vim-icinga2</option>
<command>mkdir</command> <option>-p</option> <option>~/.vim/syntax</option>
<command>ln</command> <option>-s</option> <option>/usr/share/vim/addons/syntax/icinga2.vim</option> <option>~/.vim/syntax</option></screen>
      </listitem>
    </itemizedlist>
  </tip>

  <section xml:id="sdi_icinga_sect_functional">
    <title>Functional checks</title>

    <orderedlist>
      <listitem>
        <para>Install <productname>Nagios</productname> plugins by executing
        <command>aptitude</command> <option>install</option>
        <option>nagios-plugins</option>.</para>
      </listitem>

      <listitem>
        <para>Configure a http(s) check of your web server. After successful
        configuration shut down the apache2 service and wait for error
        messeges to emerge.</para>
      </listitem>

      <listitem>
        <para>Check your local <productname>Ldap</productname> server.</para>
      </listitem>
    </orderedlist>
  </section>

  <section xml:id="sdi_icinga_sect_snmp">
    <title><xref linkend="glo_SNMP"/> based checks</title>

    <para>Use the <link
    xlink:href="http://nagios.manubulon.com/snmp_storage.html">Snmp storage
    check</link> plugin to check your root (/) file system allocation.
    Configure lower thresholds to achieve a warning message.</para>

    <para>This requires <command
    xlink:href="https://linux.die.net/man/8/snmpd">snmpd</command>
    configuration on your monitored server side:</para>

    <screen>apt-get install snmpd</screen>

    <para>File /etc/snmp/snmpd.conf might contain:</para>

    <programlisting language="none">rocommunity mi-snmp

includeAllDisks</programlisting>
  </section>

  <section xml:id="sdi_icinga2_checkBySsh">
    <title><xref linkend="glo_ssh"/> based checks</title>

    <para>Our scenario involves checking <filename>/var/log</filename> not
    exceeding a given file system size. A violation may indicate
    <abbrev>e.g.</abbrev> <command
    xlink:href="https://linux.die.net/man/8/logrotate">logrotate</command> not
    being set up properly. The subsequently described steps assume the
    <productname>Icinga</productname> software running on host
    sdi10a.mi.hdm-stuttgart.de / 141.62.75.120 querying target host
    sdi10b.mi.hdm-stuttgart.de.</para>

    <para>Target host configuration steps:</para>

    <orderedlist>
      <listitem>
        <para>Download the <filename>check_file_size.sh</filename> plugin to
        <filename>/usr/lib/nagios/plugins</filename>.</para>
      </listitem>

      <listitem>
        <para>Test <filename>check_file_size.sh</filename> manually by
        choosing appropriate warning and critical threshold values:</para>

        <screen>sdi10b#&gt; /usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn 1000 --maxcrit <emphasis
            role="red">1500</emphasis> 
FILE Critical: Size of <emphasis role="red">1610 &gt; 1500</emphasis> for /var/log

sdi10b#&gt; /usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn <emphasis
            role="red">1000</emphasis> --maxcrit 2000 
FILE Warning: Size of <emphasis role="red">1610 &gt; 1000</emphasis> for /var/log

sdi10b#&gt; /usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn 2000 --maxcrit 4000 
FILE OK: All files (1) fall within requested parameters
</screen>
      </listitem>

      <listitem xml:id="sdi_icinga_listitemRemoteExecScript">
        <para>Prepare for remote execution. Create a directory
        <filename>/etc/nagiosBySsh</filename> and the following executable
        bash script <filename>/etc/nagiosBySsh/nagioscheckssh</filename>
        within:</para>

        <programlisting language="bash">#!/bin/bash

if [ -z "$SSH_ORIGINAL_COMMAND" ]; then
    echo "Environment variable «SSH_ORIGINAL_COMMAND» undefined"
else
    exec $SSH_ORIGINAL_COMMAND
fi

#end</programlisting>

        <para>This script will later be executed by a remote <xref
        linkend="glo_ssh"/> call. The actual command to be executed will be
        provided by the environment variable
        <varname>SSH_ORIGINAL_COMMAND</varname>.</para>
      </listitem>

      <listitem>
        <para>Simulate remote execution by using <command
        xlink:href="https://linux.die.net/man/1/env">env</command> for setting
        the desired environment variable:</para>

        <screen>sdi10b#&gt; env SSH_ORIGINAL_COMMAND='/usr/lib/nagios/plugins/check_file_size.sh /var/log --maxwarn 1000 --maxcrit 1500' /etc/nagiosBySsh/nagioscheckssh
</screen>

        <para>Depending on the chosen parameter values the resulting output
        should be similar to:</para>

        <screen>FILE Critical: Size of 1610 &gt; 1500 for /var/log</screen>
      </listitem>

      <listitem>
        <para>For the sake of security a non-privileged account will be used
        for remote <xref linkend="glo_ssh"/> execution. Create a system
        account <code>nagioscheck</code> for this purpose:</para>

        <screen><command>adduser</command> <option>--shell</option> <option>/bin/bash</option> <option>--system</option> <option>nagioscheck</option></screen>
      </listitem>

      <listitem>
        <para>As user <code>nagioscheck</code> create an <xref
        linkend="glo_ssh"/> pair of keys using an empty pass phrase:</para>

        <screen>sdi10b#&gt; su - nagioscheck
sdi10b$&gt; ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/home/nagioscheck/.ssh/id_rsa): 
Created directory '/home/nagioscheck/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in <emphasis role="red">/home/nagioscheck/.ssh/id_rsa</emphasis>. <co
            xml:id="sdi_icinga_ncheckPrivateKey"/>
Your public key has been saved in <emphasis role="red">/home/nagioscheck/.ssh/id_rsa.pub</emphasis>. <co
            xml:id="sdi_icinga_ncheckPublicKey"/>
The key fingerprint is: ...</screen>
      </listitem>

      <listitem>
        <para>The private <xref linkend="glo_ssh"/> key <coref
        linkend="sdi_icinga_ncheckPrivateKey"/> will later be used for remote
        query invocations. After moving it to the querying host it may deleted
        on the target host for security reasons.</para>

        <para>Configure the corresponding public key <coref
        linkend="sdi_icinga_ncheckPublicKey"/> to allow for ordinary remote
        <xref linkend="glo_ssh"/> login:</para>

        <screen>sdi10b#&gt; su - nagioscheck
sdi10b&gt; cd ~/.ssh
sdi10b&gt; cp id_rsa.pub authorized_keys</screen>
      </listitem>

      <listitem>
        <para>We now restrict remote logins to execution of
        <filename>/etc/nagiosBySsh/nagioscheckssh</filename> from <xref
        linkend="sdi_icinga_listitemRemoteExecScript"/>. Modify
        <filename>/home/nagioscheck/.ssh/authorized_keys</filename> to
        contain:</para>

        <programlisting language="bash">from="141.62.75.110",command="/etc/nagiosBySsh/nagioscheckssh" ssh-rsa AAAAB3N...LKPFSJo5 nagioscheck@sdi10b</programlisting>

        <para>Explanation: <xref linkend="glo_ssh"/> connections origination
        from 141.62.75.110 will receive the execution result of
        <filename>/etc/nagiosBySsh/nagioscheckssh</filename>. In particular no
        login shell will be provided.</para>

        <caution>
          <para>The <option>from</option> option really requires an IP rather
          than a <xref linkend="glo_DNS"/> name.</para>
        </caution>
      </listitem>
    </orderedlist>

    <para><productname>Icinga</productname> host configuration steps:</para>

    <orderedlist>
      <listitem>
        <para>Copy the target host's
        <filename>/home/nagioscheck/.ssh/id_rsa</filename> private key to
        <filename>/etc/icinga2/nagioscheck_id_rsa</filename>. Since the
        <productname>Icinga</productname> process running with the nagios user
        is requires read access we change the ownnership accordingly:</para>

        <screen>chown nagios.nagios nagioscheck_id_rsa</screen>
      </listitem>

      <listitem>
        <para>The <productname>Icinga</productname> daemon runs with the
        effective user's id <code>nagios</code>.We thus have to assure remote
        <xref linkend="glo_ssh"/> login as user <code>nagios</code>:</para>

        <screen>ssh -i /etc/icinga2/nagioscheck_id_rsa nagioscheck@sdi10b.mi.hdm-stuttgart.de \
  "/usr/lib/nagios/plugins/check_file_size.sh /var/log \
  --maxwarn 1000 --maxcrit 1500"
The authenticity of host 'sdi10b.mi.hdm-stuttgart.de (141.62.75.120)' can't be established.
ECDSA key fingerprint is SHA256:4L5rfTIJPu1lr1gpTyvaywDE01W55roZjNGKkni/060.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'sdi10b.mi.hdm-stuttgart.de,141.62.75.120' (ECDSA) to the list of known hosts.
du: cannot read directory '/var/log/samba': Permission denied
du: cannot read directory '/var/log/unattended-upgrades': Permission denied
FILE Critical: Size of 1742 &gt; 1500 for /var/log</screen>

        <para>Explanation: Remote command execution of
        <filename>/usr/lib/nagios/plugins/check_file_size.sh</filename> as
        user <code>nagioscheck</code> lacks privileges to completely read the
        entire /var/log directory tree. This will be fixed in the next
        step.</para>
      </listitem>

      <listitem>
        <para>Turn back to your target server and allow <command
        xlink:href="https://linux.die.net/man/8/sudo">sudo</command> execution
        of <filename>/usr/lib/nagios/plugins/check_file_size.sh</filename> to
        user <code>nagioscheck</code> without providing a password. Start
        by:</para>

        <screen>sdi10b#&gt; apt-get install sudo</screen>

        <para>Subsequently create a file
        <filename>/etc/sudoers.d/nagios</filename> to contain:</para>

        <programlisting language="unset">nagioscheck     ALL=NOPASSWD : /usr/lib/nagios/plugins/check_file_size.sh</programlisting>

        <para>You should now be able to initiate <command
        xlink:href="https://linux.die.net/man/8/sudo">sudo</command>
        privileged execution from your <productname>Icinga</productname>
        host:</para>

        <screen>ssh -i /etc/icinga2/nagioscheck_id_rsa nagioscheck@sdi10b.mi.hdm-stuttgart.de \
  "<emphasis role="red">sudo</emphasis> /usr/lib/nagios/plugins/check_file_size.sh /var/log \
  --maxwarn 1000 --maxcrit 1500"
FILE Critical: Size of 1726 &gt; 1500 for /var/log</screen>
      </listitem>

      <listitem>
        <para>Configure an <productname>Icinga</productname> check command in
        <filename>conf.d/commands.conf</filename>:</para>

        <programlisting language="none">object CheckCommand "<emphasis
            role="red">by_ssh_file_size</emphasis>" {
  import "by_ssh"

  vars.by_ssh_command  = "sudo /usr/lib/nagios/plugins/check_file_size.sh --maxwarn $by_ssh_file_size_warn$ --maxcrit $by_ssh_file_size_crit$ $by_ssh_file_size_path$"
  vars.by_ssh_identity = "/etc/icinga2/nagioscheck_id_rsa"
  vars.by_ssh_logname  = "nagioscheck"

#  Parameters by_ssh_file_size_warn, vars.by_ssh_file_size_crit and
#  by_ssh_file_size_path will be defined in service or host definition
}</programlisting>
      </listitem>

      <listitem>
        <para>Define an <productname>Icinga</productname> template in
        <filename>conf.d/services.conf</filename>:</para>

        <programlisting language="none">apply Service for (path =&gt; config in <emphasis
            role="red">host.vars.paths</emphasis>) {
  import "generic-service"

  check_command = "<emphasis role="red">by_ssh_file_size</emphasis>"

  vars += config
}</programlisting>
      </listitem>

      <listitem>
        <para>Finally add two suitable host service checks in
        <filename>conf.d/hosts.conf</filename>:</para>

        <programlisting language="none">object Host "sdi10b.mi.hdm-stuttgart.de" {
  import "generic-host"

  address = "sdi10b.mi.hdm-stuttgart.de"

  vars.paths["/var/log size"] = {
    by_ssh_file_size_warn = "500"
    by_ssh_file_size_crit = "5500"
    by_ssh_file_size_path = "<emphasis role="red">/var/log</emphasis>"
  }

  vars.paths["/var/lib size"] = {
    by_ssh_file_size_warn = "50000"
    by_ssh_file_size_crit = "900000"
    by_ssh_file_size_path = "<emphasis role="red">/var/lib</emphasis>"
  }

  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }
}</programlisting>
      </listitem>
    </orderedlist>
  </section>
</chapter>