Receive SNMP Traps with Icinga 2 on Ubuntu/Debian

Configure Icinga2 to Receive SNMP Traps on Ubuntu/Debian

Warning

I’m warning you up front, making this happen is a pain if you’re new to Icinga. I did my best to account for every nuance I ran into, but you may find something else. Feel free to comment if you need help.

Introduction

Unfortunately, there is no official way for Icinga to receive SNMP traps. However, there is a pseudo official hack everyone uses to make it happen. Icinga is not meant to be a replacement for full scale SNMP management suites however, it can do a pretty good job. I originally wrote this as a word document and moved it over so the indents on the numbering below may be slightly off, but it’s all in the correct order.

Flow Synopsis

We’ll use a couple of programs in conjunction to make this happen. The device will generate SNMP traps and send them to the SNMP server, which in our example resides on the same box as Icinga. We will use snmptrapd as our SNMP trap server. We will configure the snmptrapd service with an SNMP trap handler. When snmptrapd receives an SNMP trap, it will forward the trap to the SNMP trap handler. In our case, we will use snmptt as the handler. This tool translates SNMP traps from an SNMP OID (more information here) to a meaningful set of text which represents the trap. snmptt will use an executable statement to determine what to do for that trap. Each trap will have a configurable executable statement (you can use wildcards).

In order to get Icinga to receive the alert, our executable statements in snmptt will call an Icinga event handler. This event handler will report the SNMP trap to a running Icinga service, which is what you’ll actually be able to see in Icinga itself.

Set up the SNMP Trap Daemon

Recall, the SNMP trap daemon is responsible for receiving SNMP traps from the target host.

  1. Start by installing the daemon itself with sudo apt-get install snmpd
  2. Edit the configuration for snmptrapd by running the following commands:
    1. vim /etc/snmp/snmptrapd.conf
    2. Add the following two lines. The traphandle line tells snmptrapd to feed any traps it receives to the /usr/sbin/snmptthandler program, which is part of the snmptt suite of tools we will install next. disableAuthorization yes tells snmptrapd to not screen incoming SNMP traps. You could set up snmptrapd to only receive SNMP traps from certain devices if you wanted to.
      1. traphandle default /usr/sbin/snmptthandler
      2. disableAuthorization yes
    3. The snmptrapd service script does not run properly. I haven’t taken the time to troubleshoot it, but you can run snmptrapd with snmptrapd -On -Lsd -p /var/run/snmptrapd.pid and the program will run properly.

Configure SNMPTT

SNMPTT will take the trap from snmptrapd (the trap handler) and convert it to a meaningful message which we can send to Icinga.

  1. Steps 3-5 describe manual installation of SNMPTT. This is usually not necessary. If you are not manually installing, skip to step 6. Download SNMPTT. You can download it from the command line with wget http://downloads.sourceforge.net/project/snmptt/snmptt/snmptt_1.4/snmptt_1.4.tgz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fsnmptt%2Ffiles%2F&ts=1445793160&use_mirror=superb-dca2 The file will download with a strange name, but it works if renamed to <anything>.tgz. Alternatively, download form their home page here http://snmptt.sourceforge.net/downloads.shtml.
  2. Run the following commands to install SNMPTT
    1. sudo cp snmptt snmptthandler /usr/sbin/
    2. sudo chmod +x /usr/sbin/snmptt /usr/sbin/snmptthandler
    3. sudo cp snmptt.ini /etc/snmp/
    4. sudo cp snmpttconvertmib /usr/sbin
    5. sudo groupadd snmptt
    6. sudo useradd -g snmptt snmptt
    7. sudo chown snmptt:snmptt /etc/snmp/snmptt.ini
    8. sudo mkdir /var/spool/snmptt
    9. sudo chown snmptt:snmptt /var/spool/snmptt/
    10. sudo vim /etc/snmp/snmptt.ini
      1. Change the line mode = standalone to mode=daemon
      2. If you want to change the DNS settings, change the line dns_enable = 0 to dns_enable = 1 and set strip_domain to 1
      3. Set syslog_enable to 0 if you do not have syslog set up
  1. Fix missing perl dependencies by running the following commands. SNMPTT ships with missing dependencies so they must be installed.
    1. sudo cpan install List::Util
    2. sudo cpan install Config::IniFiles
  2. Install SNMPTT by running the command sudo apt-get install snmptt
  3. Install the MIBs you would like to monitor
    1. sudo mkdir ~/.snmp
    2. sudo mkdir ~/.snmp/mibs The reason we created this folder is that we are going to use the snmptranslate tool to take the .my files we download and translate them into usable statements for the SNMPTT tool. The snmptranslate tool checks two directories for MIB files $HOME/.snmp/mibs and /usr/local/share/snmp/mibs.
    3. Download the .my files from the web. For my server, I used the Cisco MIB files, which can be downloaded from ftp://ftp.cisco.com/pub/mibs/v2/v2.tar.gz. Change to the directory above and then run wget ftp://ftp.cisco.com/pub/mibs/v2/v2.tar.gz
      1. If using the files from Cisco, simply extract the archive and then move all the .my files from the extracted folder to ~/.snmp/mibs
    4. Create a script to convert all of the MIB files to usable SNMPTT data
      1. vim snmptt-convert-script.sh
      2. Add the following lines to the script. Where it says, <YOUR-SERVICE-NAME>, this is the name of the service, which Icinga will run to receive the SNMP traps. In my case, I named the service snmp_traps. The name can be anything. This script runs the snmpttconvertmib command on every .my file in a target folder.

#!/bin/bash
for f in *.my
do
echo “Processing $f”
snmpttconvertmib –in=$f –out=/etc/snmp/snmptt.conf
–exec=’/usr/lib/nagios/plugins/submit_check_result_2 $r
<YOUR-SERVICE-NAME> 1′
done

  1. Save the following script as submit_check_result_2 in /usr/lib/Nagios/plugins/. This script is what the –exec line in the above script points to. The above script will modify snmptt.conf, which will contain a series of execution statements. These execution statements run anytime SNMPTT receives a trap, which matches the clause for the corresponding execution statement. The below script actually submits the trap to Icinga

#!/bin/sh
# SUBMIT_CHECK_RESULT
# Written by Ethan Galstad (egalstad@nagios.org)
# Last Modified: 26 Oct 15
#
# This script will write a command to the Nagios command
# file to cause Nagios to process a passive service check
# result.  Note: This script is intended to be run on the
# same host that is running Nagios.  If you want to
# submit passive check results from a remote machine, look
# at using the nsca addon.
#

# Arguments:
#  $1 = host_name (Short name of host that the service is
#       associated with)
#  $2 = svc_description (Description of the service)
#  $3 = return_code (An integer that determines the state
#       of the service check, 0=OK, 1=WARNING, 2=CRITICAL,
#       3=UNKNOWN).
#  $4 = plugin_output (A text string that should be used
#       as the plugin output for the service check)
#

echocmd=”/bin/echo”
CommandFile=”/var/run/icinga2/cmd/icinga2.cmd”
# get the current date/time in seconds since UNIX epoch
datetime=`date +%s`
# create the command line to add to the command file
cmdline=”[$datetime] PROCESS_SERVICE_CHECK_RESULT;$1;$2;$3;$4″
# append the command to the end of the command file
`$echocmd $cmdline >> $CommandFile`

  1. Run the command sudo chmod +x /usr/lib/nagios/plugins/submit_check_result_2
  2. Navigate to ~/.snmp/mibs and run the snmptt-convert-script within the folder. The script will then process all the .my files in the directory. Some may fail and depending on which MIB you downloaded that’s fine. Not all entries will be processed. You can confirm the command ran successfully by checking the file /etc/snmp/snmptt.conf. The processed entries should appear there
  3. (Optional) Add a catchall definition by adding the following lines to your snmptt.conf file:

EVENT CatchAll .1.* “SNMP Traps” Critical
FORMAT $D
EXEC /usr/local/nagios/plugins/submit_check_result_2 “$r” “snmp_traps” 2 “$O: $1 $2 $3 $4 $5”

  1. Run SNMPTT with the command: sudo /usr/sbin/snmptt –daemon –debug=1 –debugfile=/var/log/snmptt.log Note: This is only necessary for manual installs or if you do not have it installed as a service. If you installed it via aptitude, go to step 10.
  2. Run SNMPTT with sudo service snmptt start

Configure Icinga 2 to Receive the Alerts

Icinga receives the alert from SNMPTT via the command file at /var/run/icinga2/cmd/icinga2.cmd. We will use a passive service to check this file for new SNMP traps and then Icinga will report them.

  1. Edit the Icinga2 template file at /etc/icinga2/conf.d/templates.conf with vim /etc/icinga2/conf.d/templates.conf
  2. Add the following template to the file:

template Service “snmp-trap-service” {
import “generic-service”
check_command         = “passive”
enable_notifications  = 1
enable_active_checks  = 1
enable_passive_checks = 1
enable_flapping       = 0
volatile              = 1
max_check_attempts    = 1
check_interval        = 87000
enable_perfdata       = 0
vars.sla              = “24×7”
vars.dummy_state      = 2
vars.dummy_text       = “No passive check result received.”
}

apply Service “snmp_traps” {
import “snmp-trap-service”
assign where host.address
}

  1. In the snmp_traps service apply statement, the configuration applies the snmp-trap-service to all Icinga provides more details here.
  2. This configuration only works if the host name configured for the hosts object is the same as the incoming SNMP trap host name. If the two do not match, Icinga will discard the trap. On my configuration my Cisco 1721 sent a host name of 192.168.1.21 so my host name for my Icinga configuration had to be object Host “192.168.1.21″ {

25 thoughts on “Receive SNMP Traps with Icinga 2 on Ubuntu/Debian

  1. Reply

    Oliver

    Great documentation. But I have problems with the first script.

    “snmpttconvertmib –in=$f –out=/etc/snmp/snmptt.conf –exec=’/usr/lib/nagios/plugins/submit_check_result_2 $r 1′

    give me a “file not found” error back. What I´m doing wrong ?

    Thanks for your help

    Oliver

    1. Reply

      grantcurell@gmail.com Post author

      I’d need more information to tell you definitively. One of the files does not exist. Either it’s snmptt.conf, submit_check_result_2, or the script itself.

      1. Reply

        oliver

        Okay, thanks for your fast reply: You worte:

        snmpttconvertmib –in=$f –out=/etc/snmp/snmptt.conf
        –exec=’/usr/lib/nagios/plugins/submit_check_result_2 $r
        1′

        how to exchange ? If my service name is “snmp-trap” how should the file be ?
        or just simple snmpt-trap ?

        which ubuntu version are you using ?

        1. Reply

          grantcurell@gmail.com Post author

          If your service is named snmp_traps then the script would look like this:

          #!/bin/bash

          for f in *.my
          do
          echo “Processing $f”
          snmpttconvertmib –in=$f –out=etc/snmp/snmptt.conf. –exec=’/usr/lib/nagios/plugins/submit_check_result_2 $r snmp_traps 1′
          done

          1. absder

            snmpttconvertmib –in=$f –out=etc/snmp/snmptt.conf. –exec=’/usr/lib/nagios/plugins/submit_check_result_2 $r snmp_traps 1′

            In above line that dot after snmptt.conf is really needed?

  2. Reply

    theclubpenguineherald

    TRAPDRUN needs to be set to “yes” for snmptrapd service to start properly.

  3. Reply

    Oliver

    Your point 1 is getting old. With ubuntu 14.04 lts the program snmptrapd is part of the package snmpd. Please update this great documentation. Thanks a lot

    1. Reply

      grantcurell@gmail.com Post author

      Updated – thanks for the help.

  4. Reply

    Martin

    Sorry but this working to Centos 7? . Thanks

    1. Reply

      grantcurell@gmail.com Post author

      I haven’t tested it on Centos, but there’s nothing dependent on an Ubuntu specific feature in here so it should work just fine. Some of the file paths might be different or something, but I’d be surprised if it’s anything major.

  5. Reply

    MedaPusik

    All steps are clearly complete, but after I write service template to templates.conf and apply statement to services.conf, I get Syntax error. http://pastebin.com/iQ21BNm8

    1. Reply

      MedaPusik

      Well, now syntax is ok, I had to rewrite line by line manually and now it works. But if I send trap via “snmptrap -v 1 -c public 127.0.0.1 .1.3.6.1.6.3.1.1.5.1 “” 0 0 coldStart.0″ I can see trap in snmptt log but icinga is still in state “No passive check result received”. What am I doing wrong? Thank you very much for any help!

      1. Reply

        candies

        Hi, same error here. What do you mean by “I had to rewrite line by line manually” ? Did you just typed the code instead of copying it from this post? thanks for the post I helped a lot!

        1. Reply

          candies

          it helped a lot* sorry ^^

          1. Oliver

            Yes, I had to type line by line manually. Copy and paste did not work for me. I dont have a solution found. Right now we let icinga active ask the to be monitored systems.

  6. Reply

    jules chabrier

    Hi,

    Maybe some of you did know that we had to type by hand the scripts shown above in this tutorial and I didn’t.
    The characters written in this post are different than the ones you type on keyboard.

    See : http://postimg.org/image/xezv8m7zv/
    On the left, the text copied from the post, on the right it’s typed by hand.

    The result if significant for me, the snmptt_converter script seems to work well and find more OIDs to process.
    Also, when I go in /var/log/syslog I can see the translation of snmptt.

    I still doesn’t know how to push the event to icinga and I still have the “No passive check result received.” error for the check.

    Thanks for any help.

    1. Reply

      Oliver

      Hi,
      If you got an answer or have the solution please let us know. Thanks

      1. Reply

        c4ndies

        Now I will talk about the first problem : snmptt and especially snmptt.conf. When you execute the snmptt-convert-script.sh it populates the snmptt.conf with EVENTs to match.

        When snmptt receive a trap from snmptrapd, it search for a match with all the events in snmptt.conf. Once he has found one snmptt log the trap in /var/log/syslog and /var/log/snmptt/snmptt.log. The MOST IMPORTANT line of the EVENT is commented : the EXEC line.

        An EVENT has to be like this :

        EVENT linkDown .1.3.6.1.6.3.1.1.5.3 “Status Events” Normal
        FORMAT A linkDown trap signifies that the SNMP entity, acting in $*
        EXEC /usr/lib/nagios/plugins/submit_check_result_2 $A snmp_traps 2 “$Fz”
        #The EXEC calls the submit_check_result_2 to call icinga2.cmd with 4 essentials parameters :
        #$A : The host_name
        #snmp_traps : your service name in services.conf file
        #2 : Can be 0, 1 or 2. Depends of the status (OK, Warning or Critical)
        #”$Fz” : Return the “FORMAT A linkDow……” line because for me it brings the interface where the link is down.
        SDESC
        A linkDown trap signifies that the SNMP entity, acting in
        an agent role, has detected that the ifOperStatus object for
        one of its communication links is about to enter the down
        state from some other state (but not from the notPresent
        state). This other state is indicated by the included value
        of ifOperStatus.
        Variables:
        1: ifIndex
        2: ifAdminStatus
        3: ifOperStatus
        EDESC
        I hope it helped, if you want some more info please ask i will be happy to help !

  7. Reply

    jules chabrier

    Hi,

    Problem is still there. I figured out that it comes from the snmptt.conf file. I think that the “EXEC …” line which calls the “submit_cehck_result_2 script” has a problem in its syntax.

    trap –> snmptrapd = OK
    snmptrapd –> snmptt = OK
    snmptt –> submit_check_result_2 = NOT OK
    submit_check_result_2 –> Icinga2 = NOT OK

    Also, snmptrapd is now in 5.3 version. This one doesn’t accept anymore incoming traps from random community.
    If anyone understood the mecanism of access control of snmptrapd I’d be happy to learn it 🙂

    1. Reply

      jules chabrier

      Edit : Forgot to modify the snmptrapd.conf file after upgrade with “traphandle default /usr/sbin/snmptthandler
      disableAuthorization yes” content.

  8. Reply

    Hippolyth Herremans

    I have a problem with editing the templates.conf in step 15. If i do so, icinga gives an error: Backend icinga is not running.
    Any idea’s why this happens?

    1. Reply

      c4ndies

      Give your templates.conf file and services.conf file please. It’s stupid (or not?) but did you do a “service icinga2 restart” ? If you did so please give us the resut of “service icinga2 status”

  9. Reply

    Matthew Herzog

    I just love your hand sanitizer.

    1. Reply

      grantcurell@gmail.com Post author

      This is the best comment on my website.

  10. Reply

    Harald

    the script snmptt-convert-script.sh returns an error on snmpttconvertmib -> Missing arguments!
    even running the snmpttconvertmib manual gives the same error.

    running on Debian 9

Leave a Reply