Monitoring with “Non-Obvious” Nagios

John Sellens of SYONEX presents Non-Obvious Nagios.

PDF slides are available

John’s viewpoints and religion (you may not agree):
Monitoring is Exceptions, Trending, History
UNIX philosophy: Effective tools, not kitchen sink – Choose the best tool(s) for the job
SNMP is Your Friend – Use it whenever you can
Solve any problem in computer science with another level of indirection

Nagios has:
Discrete components
Well-defined interfaces
Great documentation
Nagios core just schedules and executes
It’s just an engine – that’s the simple genius of it

Well-defined and simple interface between all the parts, that’s part pf the brilliance.

He talks about compiling Nagios, and goes on to say most folks use packages. Then John mentions about monitoring status.dat, maybe with check_file_age, to make sure that Nagios is still running. And then he talks about the basics of configuration, that anyone who Nagios is familiar with.

Plugins:
Nagios Plugins or you can just write one, it’s not a hard thing to do. There are helpers like Nagios::Plugin in Perl. Then there’s more talk about plugin basics, how they work and performance data.

Put secrets in the enable-extra-opts file – they’re then hidden from ps on the server.

If a plugin does something you want, but you want a little more, you can write a wrapper around an existing script.

He goes on to talk about plugin basics, there are lots available (Nagios Exchange – 3rd party plugins), explains the difference between local/remote checks, and active/passive checks. John doesn’t use NRPE, he relies on SNMP, but that’s his self-admitted religion.

Configuration:
Talks about a bit more of the basics, especially the files themselves, that variables are case-sensitive, etc.

check_result_reaper_frequency – checks results every 10 seconds for any fork (plugin) to see if it’s finished. If you set this lower, there’s no harm [says John], and a child waiting to be reached is part of the CPU LOAD count on a unix machine – so it’s worth it, and things will finish quicker.

There’s also a variable for the frequency of checking the contents of a directory [missed which], again this can be lowered.

Set config_dir to point to where the object definitions are, and then leave it alone and put the files in there – as opposed to every time you add/remove a file, updating the cfg_file variable. The order and location of things in config files are irrelevant.

Last value of directive/variable (same thing) in a definition is used, so you can define a directive more than once, but only the last one is used. You can also inherit from templates, including multi-template inheritance (comma-separated list). Templates are usually specified with “register=0″, but you can also use a registered object as a template if you want, but it’s a bad idea – the point is either you have a template or an instantiation, and can probably abstract out if you’re in that situation.

“use” is evaluated first, no matter where in the directive it appears.

Nagios object template inheritance is a directed acyclic graph. Slide 25 is pretty useful, shows precedence and inheritance. Of course if you arrange templates such that the order doesn’t matter, then you’re golden no matter what.

You can append to an inherited value with + for instance “directive +value”. There’s no subtraction, but with host groups and hosts you can use ! for exclusion, e.g. mysql_slaves -db3

Custom directives start with an underscore (_) – case-insensitive. For example, you might want one for your SNMP community string. In your generic host template, define _snmp_community, and if there’s a different one for a specific host, you can define that in the host. And instead of “-c password”, you do “-c _snmp_community”.

Custom object variables, define a variable called “_load_warn” as 3, _load_crit as 5, the default check has that. On big machines, change that in the host definition. Refer to as macros or env variables, eg:

macro $_HOSTBLOOP$
environment variable NAGIOS__HOSTBLOOP

Implied Inheritance
Nagios will sometimes assume a value from a related object
Service objects will inherit from the associated host: contact_groups, notification_interval, notification_period
Hostescalations and serviceescalations will similarly inherit as well, except notification_period becomes escalation_period

Timeperiods are nice b/c you can exclude time periods (like holidays). There are lots of examples

Command/service definition vs. quoting – Command line quoting is sometimes challenging, so try to avoid special characters in your arguments. John’s advice, put quotes in the service definition only.

host notification options:
Don’t have “u” in things that page you, that way if things are unreachable, you won’t get paged, you want to get paged for the firewall being down or whatever. Also take off the s unless you want to be paged when scheduled downtime happens. John uses d,r.

Escalations are to a contactgroup.

Host definitions: host has a host_name and address (IP or FQDN). An IP address avoids alerts if DNS fails, but is harder to maintain. John recommends using FQDNs and having locally cached DNS on the Nagios box.

check_command is used to see if a host is up, usually it’s a ping of some kind, and it’s only checked if a service on the host fails. parents are a list of routers and gateways between Nagios and the servers. One tip is to define a “google” machine, and if Nagios can’t get to Google, all heck has broken loose b/c the whole network is gone.

hostgroups are useful for admin grouping of hosts.

Services:
In Nagios terms, a “service” could be an aspect of a running system, like disk capacity, or memory utilization. A “service” needn’t be offered externally to a device

Nagios tests services based on:
max_check_attempts — how many times to check a service before concluding it is actually down – e.g. maybe a mail queue peaks and that’s OK, but not for more than 30 min. used in conjunction with the next 2.

normal_check_interval — how many “time units” to wait between regular service checks

retry_check_interval – how many “time units” to wait before checking a service that is not “OK”

contact_groups — who to complain to in case of a problem

———
You can use notes for a URL for notes or action. Interesting, so if you have a frequent problem you could put URLs here, or an “Action url”.

dependencies & escalations are a good thing. With escalations, only add contactgroups, so that the oncall person doesn’t think it’s fixed when the manager starts getting paged.

avoid repitition:
General rule: anywhere you can list a host_name or hostgroup_name you can:
– use a comma-separated list of hosts/groups
– exclude with !
– use a wildcard host_name of “”, meaning “all hosts” to have it apply (or not) to multiple hosts

e.g. A service definition for the HTTP service might include
hostgroup_name webservers
to cause the service to be defined for all hosts in the
webservers hostgroup

Generic Notification Author (genoa) – uses environment variables.

Most are added to environment e.g. NAGIOS_SERVICESTATE including any custom variables

“On-Demand Macros” allow you to refer to values from other config settings e.g.
$SERVICESTATEID:novellserver:DS Database$

e.g. db1 is down but db2 is still up.

“On-Demand Group Macros” get you a comma-separated list of all values in a host, service or contact group e.g.
$HOSTSTATEID:hg1:,$

Documentation contains “Theory of Operation” information – read it

Then he reviews soft vs. hard state basics. state types

Stalking – If enabled, stalking logs any changes in plugin output, even with no state change, it’s logged for later review/analysis. It turns off acknowledgement, you need to ack again, because it’s a new problem.
– e.g. RAID check was “1 disk dead” and is now “2 disks dead”

Volatile services – Something that resets to OK after each check. For things that need attention every time there is a problem. Notification and event handler happen once per failure – e.g. intrusion detection system, you want to know about every time.

If you define your topology (e.g. parents) it’s easier to find the root cause of stuff.

Dependencies: Host and service dependencies define operational requirements
e.g. Web server can’t work unless file server is working

execution_failure_criteria
and
notification_failure_criteria
determine what we do
if something we depend on fails, e.g. if file server down, don’t execute web check and don’t notify me about web problem

Set inherits_parent to inherit dependencies in definitions

Cached checks
Can cache and re-use host or service check results

Used only for “On-Demand Checks” – e.g. Checking that host is up if a service fails, Checking topological reachability, For “predictive dependency checks”, Checking for “collateral damage”. Lower overhead, good results. You should enable and tune the cache.

Event Handlers:
In a perfect world, nothing would ever go wrong. In a semi-perfect world, problems would fix themselves. Event handlers are one of Nagios’ ways of moving closer to perfection.

An event handler is a command that is run in response to a state change. Canonical example: restart httpd if WWW service fails

But you could do things like open a trouble ticket on failure. You can have Global and specific host and service event handlers.

Complications: runs as the nagios user, on the nagios server.

External Commands

The Nagios server maintains a named pipe in the file system for accepting various commands from other processes. External commands are used most often by the web interface to
record information and modify Nagios’ behaviour. But you can do lots of things from shell scripts. Some of the available functionality:
– Add/delete host or service comments
– Schedule downtime, enable/disable notifications
– Reschedule host or service checks
– Submit passive service check results
– Restart or stop the Nagios server

Nagios can accept service check results from other programs. Since Nagios did not initiate the check, these are called “passive service checks”. Useful for embedded Nagios, asynchronous events, results from other, existing programs.

Nagios supports distributed monitoring of a certain style. Remote Nagios servers are essentially probe engines, submitting their results to a central server with passive service check results. The configuration on the remote servers is a subset of the central configuration. The central server is configured to notice if the passive results stop coming from the remote server.

The “central aggregation” approach is used by a number of more recent tools, such as Nagios Fusion, Thruk (slide 104), MNTOS (slide 104), and Multisite (slide 104).

Adaptive Monitoring – Can change things during runtime via external commands – e.g. schedule changes, or from an exception handler. I wonder if this could be useful for oncall rotation

Can change:
– Check commands and arguments
– Check interval, max attempts, timeperiod
– Event handler commands and arguments

Scaling Up
– Nagios can handle a lot without much effort
– As you get larger, advanced features are more important
– Use parent/child and host/service dependencies
– More efficient for humans and machines
– You will need to be more rigorous in your configuration
– Consistency, completeness, tuning
– Version 3 adds scalability and tuning features

Lots more cores now, so parallelism with plugins is more automatic.

oncall rotation in Nagios.

Tuning for Performance
– Lots of tunable configuration parameters
– Keep performance graphs of Nagios – MRTG, nagiostats, etc.
– Disable environment macros
– Use passive checks if you can- Not John’s favorite idea
– Avoid interpreted plugins, or offload checks
– Use Fast Startup Options – pre-cache configs – can save time on startup for new configs.

Tips/tricsk:
– Use the parent/child topology
– Pre Nagios 3, host checks are not parallelized
– Host checks of a down segment can block all other checks
– Be consistent and use templates and groups
– Make it easy to add another similar host
– Make it easy to add a service to a group of hosts
– Smarter plugins make life (configuration) easier (e.g. default thresholds)

With multiple Nagios servers use allow_empty_hostgroup_assignment=1 – You can define machine types as common hostgroups, even if you don’t have every type on every Nagios server. So nagios1 might not have a web server b/c it’s in an office and it won’t refuse to start nagios because a type isn’t used.

Organize Your Config Files
– Put files in different directories
– One host per config file
– Generate configs from other information you already have
– Or use a script to generate from a list
– Take advantage of your naming convention
– Wildcards in host names based on FQDNs

Contacts: sysadmin, sysadmin-email, sysadmin-page for different levels of contacting.

check_allstorage plugin – made by John – Don’t need to set limits in nagios config. Gets list of filesystems from device, cache in /tmp dir, Estimates thresholds based on current usage. NICE. nagios checks on his resources page

Web server monitoring hack – Got a visible web server that can run PHP or CGI? Set up a “hidden” web page to run your check. Use Auth or allow/deny rules to limit access. Use check_http to look for a regular expression. Get remote status over port 80.

Hosts don’t actually have to exist – you can make up a service check like “mail” that will hit a generic MX record.

Negate check – e.g. page me if ssh ever gets turned on, on this machine.

This was a great talk, I can see why it’s usually a 3-hour tutorial, and I want to take it and learn about more ideas/examples.

John Sellens of SYONEX presents Non-Obvious Nagios.

PDF slides are available

John’s viewpoints and religion (you may not agree):
Monitoring is Exceptions, Trending, History
UNIX philosophy: Effective tools, not kitchen sink Choose the best tool(s) for the job
SNMP is Your Friend Use it whenever you can
Solve any problem in computer science with another level of indirection

Nagios has:
Discrete components
Well-defined interfaces
Great documentation
Nagios core just schedules and executes
It’s just an engine thats the simple genius of it

Well-defined and simple interface between all the parts, thats part pf the brilliance.

He talks about compiling Nagios, and goes on to say most folks use packages. Then John mentions about monitoring status.dat, maybe with check_file_age, to make sure that Nagios is still running. And then he talks about the basics of configuration, that anyone who Nagios is familiar with.

Plugins:
Nagios Plugins or you can just write one, its not a hard thing to do. There are helpers like Nagios::Plugin in Perl. Then theres more talk about plugin basics, how they work and performance data.

Put secrets in the enable-extra-opts file theyre then hidden from ps on the server.

If a plugin does something you want, but you want a little more, you can write a wrapper around an existing script.

He goes on to talk about plugin basics, there are lots available (Nagios Exchange 3rd party plugins), explains the difference between local/remote checks, and active/passive checks. John doesnt use NRPE, he relies on SNMP, but thats his self-admitted religion.

Configuration:
Talks about a bit more of the basics, especially the files themselves, that variables are case-sensitive, etc.

check_result_reaper_frequency checks results every 10 seconds for any fork (plugin) to see if its finished. If you set this lower, theres no harm [says John], and a child waiting to be reached is part of the CPU LOAD count on a unix machine so its worth it, and things will finish quicker.

Theres also a variable for the frequency of checking the contents of a directory [missed which], again this can be lowered.

Set config_dir to point to where the object definitions are, and then leave it alone and put the files in there as opposed to every time you add/remove a file, updating the cfg_file variable. The order and location of things in config files are irrelevant.

Last value of directive/variable (same thing) in a definition is used, so you can define a directive more than once, but only the last one is used. You can also inherit from templates, including multi-template inheritance (comma-separated list). Templates are usually specified with register=0, but you can also use a registered object as a template if you want, but its a bad idea the point is either you have a template or an instantiation, and can probably abstract out if youre in that situation.

use is evaluated first, no matter where in the directive it appears.

Nagios object template inheritance is a directed acyclic graph. Slide 25 is pretty useful, shows precedence and inheritance. Of course if you arrange templates such that the order doesnt matter, then youre golden no matter what.

You can append to an inherited value with + for instance directive +value. Theres no subtraction, but with host groups and hosts you can use ! for exclusion, e.g. mysql_slaves -db3

Custom directives start with an underscore (_) case-insensitive. For example, you might want one for your SNMP community string. In your generic host template, define _snmp_community, and if theres a different one for a specific host, you can define that in the host. And instead of -c password, you do -c _snmp_community.

Custom object variables, define a variable called _load_warn as 3, _load_crit as 5, the default check has that. On big machines, change that in the host definition. Refer to as macros or env variables, eg:

macro $_HOSTBLOOP$
environment variable NAGIOS__HOSTBLOOP

Implied Inheritance
Nagios will sometimes assume a value from a related object
Service objects will inherit from the associated host: contact_groups, notification_interval, notification_period
Hostescalations and serviceescalations will similarly inherit as well, except notification_period becomes escalation_period

Timeperiods are nice b/c you can exclude time periods (like holidays). There are lots of examples

Command/service definition vs. quoting Command line quoting is sometimes challenging, so try to avoid special characters in your arguments. Johns advice, put quotes in the service definition only.

host notification options:
Dont have u in things that page you, that way if things are unreachable, you wont get paged, you want to get paged for the firewall being down or whatever. Also take off the s unless you want to be paged when scheduled downtime happens. John uses d,r.

Escalations are to a contactgroup.

Host definitions: host has a host_name and address (IP or FQDN). An IP address avoids alerts if DNS fails, but is harder to maintain. John recommends using FQDNs and having locally cached DNS on the Nagios box.

check_command is used to see if a host is up, usually its a ping of some kind, and its only checked if a service on the host fails. parents are a list of routers and gateways between Nagios and the servers. One tip is to define a google machine, and if Nagios cant get to Google, all heck has broken loose b/c the whole network is gone.

hostgroups are useful for admin grouping of hosts.

Services:
In Nagios terms, a “service” could be an aspect of a running system, like disk capacity, or memory utilization. A “service” needn’t be offered externally to a device

Nagios tests services based on:
max_check_attempts — how many times to check a service before concluding it is actually down e.g. maybe a mail queue peaks and thats OK, but not for more than 30 min. used in conjunction with the next 2.

normal_check_interval — how many “time units” to wait between regular service checks

retry_check_interval – how many “time units” to wait before checking a service that is not “OK”

contact_groups — who to complain to in case of a problem

You can use notes for a URL for notes or action. Interesting, so if you have a frequent problem you could put URLs here, or an Action url.

dependencies escalations are a good thing. With escalations, only add contactgroups, so that the oncall person doesnt think its fixed when the manager starts getting paged.

avoid repitition:
General rule: anywhere you can list a host_name or hostgroup_name you can:
use a comma-separated list of hosts/groups
exclude with !
use a wildcard host_name of “”, meaning “all hosts” to have it apply (or not) to multiple hosts

e.g. A service definition for the HTTP service might include
hostgroup_name webservers
to cause the service to be defined for all hosts in the
webservers hostgroup

Generic Notification Author (genoa) uses environment variables.

Most are added to environment e.g. NAGIOS_SERVICESTATE including any custom variables

“On-Demand Macros” allow you to refer to values from other config settings e.g.
$SERVICESTATEID:novellserver:DS Database$

e.g. db1 is down but db2 is still up.

“On-Demand Group Macros” get you a comma-separated list of all values in a host, service or contact group e.g.
$HOSTSTATEID:hg1:,$

Documentation contains “Theory of Operation” information read it

Then he reviews soft vs. hard state basics. Cached checks
Can cache and re-use host or service check results

Used only for “On-Demand Checks” e.g. Checking that host is up if a service fails, Checking topological reachability, For “predictive dependency checks”, Checking for “collateral damage”. Lower overhead, good results. You should enable and tune the cache.

Event Handlers:
In a perfect world, nothing would ever go wrong. In a semi-perfect world, problems would fix themselves. Event handlers are one of Nagios’ ways of moving closer to perfection.

An event handler is a command that is run in response to a state change. Canonical example: restart httpd if WWW service fails

But you could do things like open a trouble ticket on failure. You can have Global and specific host and service event handlers.

Complications: runs as the nagios user, on the nagios server.

External Commands

The Nagios server maintains a named pipe in the file system for accepting various commands from other processes. External commands are used most often by the web interface to
record information and modify Nagios’ behaviour. But you can do lots of things from shell scripts. Some of the available functionality:
– Add/delete host or service comments
– Schedule downtime, enable/disable notifications
– Reschedule host or service checks
– Submit passive service check results
– Restart or stop the Nagios server

Nagios can accept service check results from other programs. Since Nagios did not initiate the check, these are called “passive service checks”. Useful for embedded Nagios, asynchronous events, results from other, existing programs.

Nagios supports distributed monitoring of a certain style. Remote Nagios servers are essentially probe engines, submitting their results to a central server with passive service check results. The configuration on the remote servers is a subset of the central configuration. The central server is configured to notice if the passive results stop coming from the remote server.

The “central aggregation” approach is used by a number of more recent tools, such as Nagios Fusion, Thruk (slide 104), MNTOS (slide 104), and Multisite (slide 104).

Adaptive Monitoring Can change things during runtime via external commands e.g. schedule changes, or from an exception handler. I wonder if this could be useful for oncall rotation

Can change:
– Check commands and arguments
– Check interval, max attempts, timeperiod
– Event handler commands and arguments

Scaling Up
– Nagios can handle a lot without much effort
– As you get larger, advanced features are more important
– Use parent/child and host/service dependencies
– More efficient for humans and machines
– You will need to be more rigorous in your configuration
– Consistency, completeness, tuning
– Version 3 adds scalability and tuning features

Lots more cores now, so parallelism with plugins is more automatic.

oncall rotation in Nagios.

Tuning for Performance
– Lots of tunable configuration parameters
– Keep performance graphs of Nagios MRTG, nagiostats, etc.
– Disable environment macros
– Use passive checks if you can- Not Johns favorite idea
– Avoid interpreted plugins, or offload checks
– Use Fast Startup Options – pre-cache configs can save time on startup for new configs.

Tips/tricsk:
– Use the parent/child topology
– Pre Nagios 3, host checks are not parallelized
– Host checks of a down segment can block all other checks
– Be consistent and use templates and groups
– Make it easy to add another similar host
– Make it easy to add a service to a group of hosts
– Smarter plugins make life (configuration) easier (e.g. default thresholds)

With multiple Nagios servers use allow_empty_hostgroup_assignment=1 You can define machine types as common hostgroups, even if you don’t have every type on every Nagios server. So nagios1 might not have a web server b/c its in an office and it wont refuse to start nagios because a type isnt used.

Organize Your Config Files
– Put files in different directories
– One host per config file
– Generate configs from other information you already have
– Or use a script to generate from a list
– Take advantage of your naming convention
– Wildcards in host names based on FQDNs

Contacts: sysadmin, sysadmin-email, sysadmin-page for different levels of contacting.

check_allstorage plugin made by John Don’t need to set limits in nagios config. Gets list of filesystems from device, cache in /tmp dir, Estimates thresholds based on current usage. NICE. nagios checks on his resources page

Web server monitoring hack Got a visible web server that can run PHP or CGI? Set up a “hidden” web page to run your check. Use Auth or allow/deny rules to limit access. Use check_http to look for a regular expression. Get remote status over port 80.

Hosts dont actually have to exist you can make up a service check like mail that will hit a generic MX record.

Negate check e.g. page me if ssh ever gets turned on, on this machine.

This was a great talk, I can see why its usually a 3-hour tutorial, and I want to take it and learn about more ideas/examples.