LSM, SELinux, Netlabel, and CIPSO

LSM, SELinux, Netlabel, and CIPSO

If you are new to CIPSO and SELinux, their relationship is confusing to say the least. I wanted to shed some light for those just setting out on the topic as to how the four of these things work together.

LSM

Original paper describing LSM: Linux Security Module Framework.

Within the context of SELinux and Netlabel you may hear the term LSM domain, LSM user, LSM xxxx. Early on in the development of Linux users realized discretionary access control (DAC) did not provide sufficient security for the system. While convenient and straightforward, simple problems existed. If a process running as root is compromised, essentially the entire system is compromised. Enter mandatory access control (MAC) implemented by LSM. I won’t cover MAC here, but it provides a solution to the problems presented by DAC. Practitioners of Linux began building kernel modules to implement MAC, but the issue was there were many and none were standardized. LSM provides a solution to this problem by offering a common framework for structuring and building kernel security modules. SELinux is one such implementation compliant with the LSM model. There are many others so when you read the term LSM xxxx, what it is saying is the implementation of LSM xxxx for your specific kernel security module.  For example, LSM domain in SELinux simply refers to SELinux Domains. The name doesn’t have to directly correspond, but it typically does.

SELinux

Secure Linux is a kernel security module which implements the Bell-LaPadula MAC model. SELinux is an LSM framework compliant Linux kernel security module. It provides mechanisms for controlling the way in which users, processes, and files within a system interact. Most relevant to this post, it allows administrators to classify files, processes, users, etc according to a security classification scheme. Ex: Top secret, secret, unclassified, etc.

Multi-Level Security

Built into SELinux is something called MLS (Multi-Level Security). This enables Linux to label files according to security sensitivities with s0 being the lowest classification and s15 being the highest. Within each security sensitivity are categories which may range from 0-1024. The security sensitivity and category are combined to create what is called a security level. It is the security level that CIPSO transports to other devices on the network.

Netlabel

Netlabel is a toolset for supporting network-based labeling within SELinux. The MLS portion of SELinux creates and attaches labels to different objects within the system. Netlabel provides the means to affix security related labeling to packets in a CIPSO format.

CIPSO

Commercial Internet Protocol Security Option (CIPSO) is a generic specification for adding security levels to network traffic. Technically, it was never made a standard and remains an IETF draft, but is the defacto industry standard. SELinux passes security classification labels to the network stack and those security classification labels are translated to the CIPSO protocol and then sent across the network to other MLS enabled systems.

Setting Up SELinux (A quickstart)

What This Tutorial Covers

The aim of this tutorial is to get the user up and running with SELinux quickly and provide additional resources for further research.

Tutorial performed on: CentOS-7 with kernel version 3.10.0-514.el7.x86_64

Installing SELinux

Note: For this tutorial, all commands are run as root.

Start by installing all necessary SELinux files:

yum install policycoreutils policycoreutils-python selinux-policy selinux-policy-targeted libselinux-utils setroubleshoot-server setools setools-console mcstrans netlabel_tools selinux-policy-mls

Configuring MLS

Enable MLS

For a more in depth explanation of MLS concepts see here.

Next we must enable MLS. SELinux enables MLS via the /etc/selinux/config file. We suggest users change the SELinux mode to permissive. SELinux enforcing mode blocks connections and may prevent the operating system from booting. Change the SELINUXTYPE to MLS.

vim /etc/selinux/config


# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=mls

Now we need to instruct the system to relabel all files according to a default labeling scheme. This is accomplished by instructing the system to autorelabel everything. Do this by adding the .autorelabel file to the root directory.

touch ./autorelabel

Now reboot the system. It may take a bit for the system to relabel everything. You should see the following message while it is relabeling:

Once the system reboots you can check to ensure it is running in permissive mode and  that MLS is enabled:

sestatus

SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: mls
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: denied
Max kernel policy version: 28

Warning: Most setups have a number of errors generated when MLS is enabled. MLS likely blocks a number of critical processes. I suggest you resolve these errors before swapping from permissive to enforcing.

You can check the logs to see the activity SELinux blocks:

cat /var/log/audit/audit.log | grep SEL

If nothing appears when you run the command you are likely safe to swap to enforcing assuming the system has performed all of its expected functionality.

Configuring MLS Security Levels

To configure the security levels for files and processes edit the file /etc/selinux/mls/setrans.conf:

#
# Multi-Level Security translation table for SELinux
#
# Uncomment the following to disable translation libary
# disable=1
#
# Objects can be labeled with one of 16 levels and be categorized with 0-1023
# categories defined by the admin.
# Objects can be in more than one category at a time.
# Users can modify this table to translate the MLS labels for different purpose.
#
# Assumptions: using below MLS labels.
#  SystemLow
#  SystemHigh
#  Unclassified
#  Secret with compartments A and B.
#
# SystemLow and SystemHigh
s0=SystemLow
s15:c0.c1023=SystemHigh
s0-s15:c0.c1023=SystemLow-SystemHigh

# Unclassified level
s1=Unclassified

# Secret level with compartments
s2=Secret
s2:c0=A
s2:c1=B

# ranges for Unclassified
s0-s1=SystemLow-Unclassified
s1-s2=Unclassified-Secret
s1-s15:c0.c1023=Unclassified-SystemHigh

# ranges for Secret with compartments
s0-s2=SystemLow-Secret
s0-s2:c0=SystemLow-Secret:A
s0-s2:c1=SystemLow-Secret:B
s0-s2:c0,c1=SystemLow-Secret:AB
s1-s2:c0=Unclassified-Secret:A
s1-s2:c1=Unclassified-Secret:B
s1-s2:c0,c1=Unclassified-Secret:AB
s2-s2:c0=Secret-Secret:A
s2-s2:c1=Secret-Secret:B
s2-s2:c0,c1=Secret-Secret:AB
s2-s15:c0.c1023=Secret-SystemHigh
s2:c0-s2:c0,c1=Secret:A-Secret:AB
s2:c0-s15:c0.c1023=Secret:A-SystemHigh
s2:c1-s2:c0,c1=Secret:B-Secret:AB
s2:c1-s15:c0.c1023=Secret:B-SystemHigh
s2:c0,c1-s15:c0.c1023=Secret:AB-SystemHigh

For an in depth explanation of configuring the security levels see this power point. Page 42 includes a sample configuration for a simple US DOD style classification system. You can print a quick listing of the security levels with the chcat -L command.

Configuring Users and Files for MLS

Next we need to add a system user to the list of SELinux users. SELinux maps the system users to SELinux users. SELinux users map to roles. The roles define which users may access what processes. Processes have permissions to act on specific objects IE: files, directories, TCP sockets, etc. This tutorial does a good job of explaining these concepts. See the section titled “Some Basic Terminology”. Now we will add a user:

semanage login --add --seuser user_u grant

user_u is what is called a type in SELinux terminology. In this case we want to add a user of user_u type. grant is the system user we want to add, or at least, I want to add :-D.

To view the user mappings you can use the list command within the semanage tool.

semanage login -l

This mapping also shows us what MLS range users have access to. In this case, we can see the user grant (me) only has access to the s0 security sensitivity. We can edit a user’s security access levels with the following command:

semanage login –modify –seuser user_u –range s0-s2 grant

If user_u has not been configured, you will likely receive this error:

[root@localhost ~]# semanage login --modify --seuser user_u --range s2:c100 grant
libsemanage.validate_handler: MLS range s2:c100 for Unix user grant exceeds allowed range s0 for SELinux user user_u (No such file or directory).
libsemanage.validate_handler: seuser mapping [grant -> (user_u, s2:c100)] is invalid (No such file or directory).
libsemanage.dbase_llist_iterate: could not iterate over records (No such file or directory).
OSError: No such file or directory

What this is telling you is that the user_u SELinux user does not have access to either s1 or s2 or both (you don’t know which). To fix this you must grant the user_u SELinux user permissions to those security sensitivities first.

semanage user --modify --range s0-s2 user_u

Now we can rerun the semanage login command and it should go through.

Finally, we have to set the permissions for files within the file system. First, it is helpful to be able to check the permissions on the file system. In most Linux commands which support SELinux, the security permissions of an object are viewable with the -Z option. Ex: ps -Z, ls -Z, etc.

ls -Z

WARNING: SELinux permissions do not supersede the regular DACL permissions set by Linux. DACL permissions are checked first and only after that check does the system consult SELinux. This means if DACL denies a user access to a file and SELinux permits it, the user will not be able to access the file.

Right now we see that test_file has a permission level of s0. If we wanted to make it so that user grant could not access the file we could change the file’s security sensitivity to s3.

Note: In SELinux processes may write to their own security level or up and may read their security level and down. The process cannot read up or write down.

You can set the SELinux file permissions with the chcon utility.

chcon -l s3 test_file

Now we see the permissions of test_file require s3 access.

Conclusion

Hope this tutorial saves everyone some time! If you have questions feel free to comment.

Helpful Commands

Edit SELinux Configuration File

vim /etc/selinux/config

Check SELinux Status

sestatus

Check logs for SELinux activity:

cat /var/log/audit/audit.log | grep SEL

Configure MLS Security Levels

vim /etc/selinux/mls/setrans.conf

Print a listing of the security levels:

chcat -L

Add a User to SELinux

semanage login --add --seuser user_u grant

List SELinux User Mappings and Permissions

semanage login -l

Grant permissions to an SELinux user for a specific security sensitivity range:

semanage user --modify --range s0-s2 user_u

Grant permissions to a system user for a specific security sensitivity range:

semanage login --modify --seuser user_u --range s0-s2 grant

List file or process SELinux permissions:

ls -Z
ps -Z

Change the SELinux file permissions:

chcon -l s3 test_file

 

A Simple SSL Client and Server in Python

Server

import socket
from socket import AF_INET, SOCK_STREAM, SO_REUSEADDR, SOL_SOCKET, SHUT_RDWR
import ssl

KEYFILE = 'server_key.pem'
CERTFILE = 'server_cert.pem'

def echo_client(s):
    while True:
        data = s.recv(8192)
        print(data.decode("utf-8"))
        if data == b'':
            break
        s.send(b'This is a response.')
        print('Connection closed')
    s.close()

def echo_server(address):
    s = socket.socket(AF_INET, SOCK_STREAM)
    s.bind(address)
    s.listen(1)
    s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)

    s_ssl = ssl.wrap_socket(s, keyfile=KEYFILE, certfile=CERTFILE, server_side=True)

    while True:
        try:
            (c,a) = s_ssl.accept()
            print('Got connection', c, a)
            echo_client(c)
        except socket.error as e:
            print('Error: {0}'.format(e))

echo_server((socket.gethostbyname('localhost'), 8082))

Client

import ssl

port = 8082

import socket, ssl

while True:

    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Require a certificate from the server. We used a self-signed certificate
    # so here ca_certs must be the server certificate itself.
    ssl_sock = ssl.wrap_socket(s,cert_reqs=ssl.CERT_REQUIRED, ca_certs='server_cert.pem')

    ssl_sock.connect(('127.0.0.1', 8082))

    ssl_sock.write(str(input("Enter Something: ")).encode())

    ssl_sock.close()

Pentesting: Runas – How to elevate the same command prompt from user to admin in Windows

The search that’s more likely to bring you here might be something to the effect of: how to elevate the same command prompt from user to admin in Windows. For me, it was surprisingly annoying and I wasted a lot more time than I thought I would tracking down a solution to this particular problem.

It’s a deceptively simple task and the short answer to your question is: depending on the scenario, you can’t – but there are ways around it. As the title eludes, this is unfortunately not a post about how to elevate your privileges in the same command prompt legitimately, but how to do so in a pentesting scenario. (Although if you are a legit admin, you can set things up so you can do it with psexec.)

Scenario

You are pentesting something and you can get command line access to a server as a user. You have administrative credentials, but you only have access to your one user shell and you want to elevate to administrator/system.

How to Do It

Fortunately, it’s pretty straight forward. First thing you have to do is get an a meterpreter shell running on the computer. You may run into AV and if that’s the case you’ll need to obfuscate your meterpreter somehow. Fortunately this is pretty easy to do, something like Veil will do the trick, but really any non-Metasploit crypto/obfuscation typically works. All the AV vendors have signatures for the built in Metasploit stuff so it tends to be fairly ineffective.

Once you have your meterpreter session running you can use a module called post/windows/manage/run_as. The options will look something like this:

CMD <YOUR_METERPRETER_PAYLOAD> yes Command to execute
CMDOUT false yes Retrieve command output
DOMAIN workgroup yes Domain to login with
PASSWORD <ADMIN_PASSWORD> yes Password to login with
SESSION <YOUR_USER_METERPRETER_SESSION> yes The session to run this module on.
USER <ADMIN_USERNAME> yes Username to login with

Where CMD could be anything you want to run as the administrator, but I typically just rerun my meterpreter payload to upgrade it to admin level and go from there.

How to: Unicode in URL with Python 3

I found this to be much more difficult than I thought it would be. The solution was simple, but finding it was a bit of a pain. Here is the solution:

from urllib.parse import quote
from urllib.request import urlopen

url_string = "http://your_site_here" + quote(SOME-STRING-WITH-UNICODE)
html = urlopen(url_string).read().decode('utf-8')

Extremely simple, but it caused me a fair amount of headache to figure that out.

Common errors that might bring you to this post:

AttributeError: ‘bytes’ object has no attribute ‘timeout’
TypeError: Can’t convert ‘bytes’ object to str implicitly
UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xf3’ in position 6: ordinal not in range(128)
TypeError: a bytes-like object is required, not ‘str’
TypeError: expected bytes-like object, not str

 

Fixing asm: Internal error: unbalanced parenthesis in operand 1

I got this error the other day when I was trying to compile the exploit code for the mremap exploit on Linux. While I have programmed a lot I have not spent an inordinate amount of time with gcc and this was something of a pain to solve so I thought I would share a general solution.

1.) Begin by getting a dump of the assembly code. You can do this with gcc -S -o <name_of_output> <name_of_input>

2.) Go to the line the compiler told you there was an issue with. Now you can see what the problem is. In my case it was that the assembler came up with: movl $((17 which obviously has missing parenthesis.

3.) You can either directly modify the assembly, or trace it back to the root file. In my case, it was inline assembly and doing a search on the corresponding function name led me to the offending line: ” movl $(“xstr(CLONEFL)”), %%ebx \n”. It was a macro, which looked fine. However you can view preprocessor output with the -E option. Examining the preprocessor output led me to the line: ”               movl $(“”(17|0x00004000|0x00000100)””), %%ebx           \n”. I’m not sure why that doesn’t work, but I simply changed it to ”               movl $0x4111, %%ebx                     \n” and it worked.

Hope this saves someone some time. Feel free to comment if you have more specific questions.

Connect GNS3 to ESXi

Introduction

As far as I can tell, there’s no great way to make this happen. I will explain, but to give you an image up front, below is a diagram of what I did. It may seem daunting at first, but I’ll explain as we go along.

gns3-esxi-diagram

Set Up Description

My GNS3 server is running on a VM on my ESXi host. In my case, I was running GNS3 on top of Ubuntu 15.10. My GNS3 server has four interfaces relevant to this problem. Interface eno16777984 connects from the VM to the default vSwitch0, which has access to the ESXi server’s one real network interface card. Interface eno33557248 connects to a second virtual switch I created, vSwitch1. This is the switch to which I connected the virtual machines I wanted to connect into my GNS3 topology. The interface tap1 is a loopback interface on the GNS3 server, which I used to connect into my GNS3 topology. Interface br0 bridges the tap1 interface to interface eno33557248. The bridge connects the virtual network created by vSwitch1 as bridged to my GNS3 topology.

Flow summary: VM hosted on ESXi -> vSwitch1 -> eno33557248 (GNS3 server)-> br0 (GNS3 server) -> tap1 (GNS3 server) -> GNS3 cloud

Apology: Sorry about the wonky interface names. Not sure why ESXi causes Ubuntu to generate such bizarre names.

Limitations

The limitation of this solution is that you may have to implement it multiple times if you want to connect different ESXi VMs into different locations within your GNS3 topology. For example, vSwitch1 could be used to service all the DMZ machines in a GNS3 topology. However, if you want to plug the ESXi VMs into a different location, say at the access layer, you will need to set up another iteration of this solution in its totality.

Configure ESXi Server

  1. Select your ESXi server, go to configuration->Networking
  2. Click Add Networking, Virtual Machine, Create a vSphere standard switch, label it and put it in a VLAN – I used VLAN 2, click finish
  3. On your GNS3 server, add a virtual NIC which is connected to your newly created vSwitch
  4. On your newly created vSwitch go to properties, highlight your newly created network (the one you named – not the one that says vSwitch), click edit, go to security and then click the checkbox next to promiscuous mode and change the setting to accept. This setting is not ideal and this technique should not be used in production networks. It essentially turns the vSwitch into a hub. I didn’t delve into depth on the issue, but I noticed vSwitch does not handled bridge traffic properly. It will forward layer two traffic, but layer 3. This setting is required for the bridge we create later on to work. As best as I can tell, it looks like vSwitch doesn’t learn the MAC addresses from the other network. So it doesn’t forward destination traffic properly.

esxi

My final configuration looked like this:

esxi-configuration

Configure GNS3 Server Interfaces

This is the tricky part of the operation. Credit goes to knowosielski for his post here for illustrating how to connect an interface into the GNS3 topology.

Create a Virtual Interface

  1. Create a shell script with the following content and put it in your location of choice (i.e. /scripts/<SCRIPT NAME>)

#!/bin/bash

#Set TAP1
tunctl -u husband

#Configure TAP1
ifconfig tap1 up

WARNING: Your tap interface may come up as tap0. That’s fine. When I set this up, I already had a tap0 so mine came up with tap1. If yours comes up as tap0, simply adjust the following steps accordingly.

  1. Modify the line “tunctl -u husband” and replace “husband” with your user name you want to have access to the interface.
  2. Save the script and make it executable with chmod +x <SCRIPT_NAME>
  3. Test the script by running it, then do an ifconfig and make sure tap1 is there. (Reminder, yours may come up as tap0, adjust steps accordingly if this is the case.)
  4. Modify “/etc/rc.local” to run this script every time the system starts. Add the line sudo <PATH_TO_SCRIPT>/<NAME_OF_SCRIPT> BEFORE the line exit 0. If you do not add the line before exit 0 it will not work. In mine I added the line sudo /home/husband/GNS3/script/tap
  5. Consider testing to make sure everything works by rebooting the system.

Create the Bridge Interface

  1. If you don’t already have them, run sudo apt-get install bridge-utils
  2. sudo vim /etc/network/interfaces add the line auto eno33557248 or whatever the name is of your VMs second interface. This should be the interface which resides in your newly created virtual ESXi virtual network which in my case was on vSwitch1.
  3. Now add the following lines:

# Bridge between tap1 and eno33557248
auto br0
iface br0 inet manual
bridge_ports tap1 eno33557248
bridge_stp off

  1. At this juncture, I strongly recommend you do a reboot and make sure that everything works at this point. If you skip this, troubleshooting down the line will probably be more challenging.

Configure GNS3

Now we’ll configure GNS3 itself. My setup was very simple for the sake of making sure everything works:

gns3_side

  1. In GNS3 click “Browse all devices” and drop the cloud into your topology
  2. Right click on the cloud and select configure
  3. Go to the tap tab, type tap1 (or whatever your tap interface is named), and click add

gns3

Now just drop a device in and connect it to the cloud and you should be up and running. I also tested this with the GNS3 ethernet switch and it worked fine. See screenshot below. This is a separate Ubuntu 15.10 server residing on my newly created ESXi vSwitch1, pinging through my GNS3 server and into the GNS3 topology.

gns32

This set up took me a really long time (especially that bit with promiscuous mode – that took forever to figure out). If you have any questions feel free to comment.

Receive SNMP Traps with Icinga 2 on Ubuntu/Debian

Configure Icinga2 to Receive SNMP Traps on Ubuntu/Debian

Warning

I’m warning you up front, making this happen is a pain if you’re new to Icinga. I did my best to account for every nuance I ran into, but you may find something else. Feel free to comment if you need help.

Introduction

Unfortunately, there is no official way for Icinga to receive SNMP traps. However, there is a pseudo official hack everyone uses to make it happen. Icinga is not meant to be a replacement for full scale SNMP management suites however, it can do a pretty good job. I originally wrote this as a word document and moved it over so the indents on the numbering below may be slightly off, but it’s all in the correct order.

Flow Synopsis

We’ll use a couple of programs in conjunction to make this happen. The device will generate SNMP traps and send them to the SNMP server, which in our example resides on the same box as Icinga. We will use snmptrapd as our SNMP trap server. We will configure the snmptrapd service with an SNMP trap handler. When snmptrapd receives an SNMP trap, it will forward the trap to the SNMP trap handler. In our case, we will use snmptt as the handler. This tool translates SNMP traps from an SNMP OID (more information here) to a meaningful set of text which represents the trap. snmptt will use an executable statement to determine what to do for that trap. Each trap will have a configurable executable statement (you can use wildcards).

In order to get Icinga to receive the alert, our executable statements in snmptt will call an Icinga event handler. This event handler will report the SNMP trap to a running Icinga service, which is what you’ll actually be able to see in Icinga itself.

Set up the SNMP Trap Daemon

Recall, the SNMP trap daemon is responsible for receiving SNMP traps from the target host.

  1. Start by installing the daemon itself with sudo apt-get install snmpd
  2. Edit the configuration for snmptrapd by running the following commands:
    1. vim /etc/snmp/snmptrapd.conf
    2. Add the following two lines. The traphandle line tells snmptrapd to feed any traps it receives to the /usr/sbin/snmptthandler program, which is part of the snmptt suite of tools we will install next. disableAuthorization yes tells snmptrapd to not screen incoming SNMP traps. You could set up snmptrapd to only receive SNMP traps from certain devices if you wanted to.
      1. traphandle default /usr/sbin/snmptthandler
      2. disableAuthorization yes
    3. The snmptrapd service script does not run properly. I haven’t taken the time to troubleshoot it, but you can run snmptrapd with snmptrapd -On -Lsd -p /var/run/snmptrapd.pid and the program will run properly.

Configure SNMPTT

SNMPTT will take the trap from snmptrapd (the trap handler) and convert it to a meaningful message which we can send to Icinga.

  1. Steps 3-5 describe manual installation of SNMPTT. This is usually not necessary. If you are not manually installing, skip to step 6. Download SNMPTT. You can download it from the command line with wget http://downloads.sourceforge.net/project/snmptt/snmptt/snmptt_1.4/snmptt_1.4.tgz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fsnmptt%2Ffiles%2F&ts=1445793160&use_mirror=superb-dca2 The file will download with a strange name, but it works if renamed to <anything>.tgz. Alternatively, download form their home page here http://snmptt.sourceforge.net/downloads.shtml.
  2. Run the following commands to install SNMPTT
    1. sudo cp snmptt snmptthandler /usr/sbin/
    2. sudo chmod +x /usr/sbin/snmptt /usr/sbin/snmptthandler
    3. sudo cp snmptt.ini /etc/snmp/
    4. sudo cp snmpttconvertmib /usr/sbin
    5. sudo groupadd snmptt
    6. sudo useradd -g snmptt snmptt
    7. sudo chown snmptt:snmptt /etc/snmp/snmptt.ini
    8. sudo mkdir /var/spool/snmptt
    9. sudo chown snmptt:snmptt /var/spool/snmptt/
    10. sudo vim /etc/snmp/snmptt.ini
      1. Change the line mode = standalone to mode=daemon
      2. If you want to change the DNS settings, change the line dns_enable = 0 to dns_enable = 1 and set strip_domain to 1
      3. Set syslog_enable to 0 if you do not have syslog set up
  1. Fix missing perl dependencies by running the following commands. SNMPTT ships with missing dependencies so they must be installed.
    1. sudo cpan install List::Util
    2. sudo cpan install Config::IniFiles
  2. Install SNMPTT by running the command sudo apt-get install snmptt
  3. Install the MIBs you would like to monitor
    1. sudo mkdir ~/.snmp
    2. sudo mkdir ~/.snmp/mibs The reason we created this folder is that we are going to use the snmptranslate tool to take the .my files we download and translate them into usable statements for the SNMPTT tool. The snmptranslate tool checks two directories for MIB files $HOME/.snmp/mibs and /usr/local/share/snmp/mibs.
    3. Download the .my files from the web. For my server, I used the Cisco MIB files, which can be downloaded from ftp://ftp.cisco.com/pub/mibs/v2/v2.tar.gz. Change to the directory above and then run wget ftp://ftp.cisco.com/pub/mibs/v2/v2.tar.gz
      1. If using the files from Cisco, simply extract the archive and then move all the .my files from the extracted folder to ~/.snmp/mibs
    4. Create a script to convert all of the MIB files to usable SNMPTT data
      1. vim snmptt-convert-script.sh
      2. Add the following lines to the script. Where it says, <YOUR-SERVICE-NAME>, this is the name of the service, which Icinga will run to receive the SNMP traps. In my case, I named the service snmp_traps. The name can be anything. This script runs the snmpttconvertmib command on every .my file in a target folder.

#!/bin/bash
for f in *.my
do
echo “Processing $f”
snmpttconvertmib –in=$f –out=/etc/snmp/snmptt.conf
–exec=’/usr/lib/nagios/plugins/submit_check_result_2 $r
<YOUR-SERVICE-NAME> 1′
done

  1. Save the following script as submit_check_result_2 in /usr/lib/Nagios/plugins/. This script is what the –exec line in the above script points to. The above script will modify snmptt.conf, which will contain a series of execution statements. These execution statements run anytime SNMPTT receives a trap, which matches the clause for the corresponding execution statement. The below script actually submits the trap to Icinga

#!/bin/sh
# SUBMIT_CHECK_RESULT
# Written by Ethan Galstad (egalstad@nagios.org)
# Last Modified: 26 Oct 15
#
# This script will write a command to the Nagios command
# file to cause Nagios to process a passive service check
# result.  Note: This script is intended to be run on the
# same host that is running Nagios.  If you want to
# submit passive check results from a remote machine, look
# at using the nsca addon.
#

# Arguments:
#  $1 = host_name (Short name of host that the service is
#       associated with)
#  $2 = svc_description (Description of the service)
#  $3 = return_code (An integer that determines the state
#       of the service check, 0=OK, 1=WARNING, 2=CRITICAL,
#       3=UNKNOWN).
#  $4 = plugin_output (A text string that should be used
#       as the plugin output for the service check)
#

echocmd=”/bin/echo”
CommandFile=”/var/run/icinga2/cmd/icinga2.cmd”
# get the current date/time in seconds since UNIX epoch
datetime=`date +%s`
# create the command line to add to the command file
cmdline=”[$datetime] PROCESS_SERVICE_CHECK_RESULT;$1;$2;$3;$4″
# append the command to the end of the command file
`$echocmd $cmdline >> $CommandFile`

  1. Run the command sudo chmod +x /usr/lib/nagios/plugins/submit_check_result_2
  2. Navigate to ~/.snmp/mibs and run the snmptt-convert-script within the folder. The script will then process all the .my files in the directory. Some may fail and depending on which MIB you downloaded that’s fine. Not all entries will be processed. You can confirm the command ran successfully by checking the file /etc/snmp/snmptt.conf. The processed entries should appear there
  3. (Optional) Add a catchall definition by adding the following lines to your snmptt.conf file:

EVENT CatchAll .1.* “SNMP Traps” Critical
FORMAT $D
EXEC /usr/local/nagios/plugins/submit_check_result_2 “$r” “snmp_traps” 2 “$O: $1 $2 $3 $4 $5”

  1. Run SNMPTT with the command: sudo /usr/sbin/snmptt –daemon –debug=1 –debugfile=/var/log/snmptt.log Note: This is only necessary for manual installs or if you do not have it installed as a service. If you installed it via aptitude, go to step 10.
  2. Run SNMPTT with sudo service snmptt start

Configure Icinga 2 to Receive the Alerts

Icinga receives the alert from SNMPTT via the command file at /var/run/icinga2/cmd/icinga2.cmd. We will use a passive service to check this file for new SNMP traps and then Icinga will report them.

  1. Edit the Icinga2 template file at /etc/icinga2/conf.d/templates.conf with vim /etc/icinga2/conf.d/templates.conf
  2. Add the following template to the file:

template Service “snmp-trap-service” {
import “generic-service”
check_command         = “passive”
enable_notifications  = 1
enable_active_checks  = 1
enable_passive_checks = 1
enable_flapping       = 0
volatile              = 1
max_check_attempts    = 1
check_interval        = 87000
enable_perfdata       = 0
vars.sla              = “24×7”
vars.dummy_state      = 2
vars.dummy_text       = “No passive check result received.”
}

apply Service “snmp_traps” {
import “snmp-trap-service”
assign where host.address
}

  1. In the snmp_traps service apply statement, the configuration applies the snmp-trap-service to all Icinga provides more details here.
  2. This configuration only works if the host name configured for the hosts object is the same as the incoming SNMP trap host name. If the two do not match, Icinga will discard the trap. On my configuration my Cisco 1721 sent a host name of 192.168.1.21 so my host name for my Icinga configuration had to be object Host “192.168.1.21″ {

Icinga “ did not exit properly error”

I got this and it took some time to troubleshoot. The error is misleading. What it’s really telling you is that the plugin failed to execute properly. Here’s the catch, this includes warnings. So if you manually test the plugin from the command line outside of Icinga and it works, but gives a warning, when you run it with Icinga it will fail and throw the aforementioned error. To fix the problem, you have to fix whatever is in the code giving the warning.

Netint Plugin throws did not exit properly error

This was the specific problem I was having trouble with. I ran it on the command line and got some errors about an uninitialized variable. On line 2023  you’ll see a line with the variable $oid_perf_inoct. That is the offending variable. You need to add an additional if statement outside this one. Enclose the whole thing in an if block of

if (defined $oid_perf_inoct[$I]) {
//All that other stuff from the if block on line 2023
}

It worked for me after that.