Nagios plugin to count apache threads

Overview

At work I have a misbehaving web server. Sometimes it spawns the maximum number of apache threads (which has a hardcoded maximum of 256, no matter what you configure) and then occupies 100% of the processor. I have decided that the normal nagios checks for the http site and ssh and so on aren’t good enough for monitoring purposes.

So I wrote my own simple nagios check. And then I put it in an rpm for easy deployment.

The nagios check

Here is the code for check_apache_threads, although you can check the latest version at my github page.

#!/bin/sh
# File: /usr/lib64/nagios/plugins/check_apache_threads
# Author: bgstack15@gmail.com
# Startdate: 2017-01-09 15:53
# Title: Nagios Check for Apache Threads
# Purpose: For a troublesome dmz wordpress host
# Package: nagios-plugins-apache-threads
# History:
# Usage:
# In nagios/nconf, use this checkcommand check command line: $USER1$/check_by_ssh -H $HOSTADDRESS$ -C "$USER1$/check_apache_threads -w $ARG1$ -c $ARG2$"
# Reference: general design /usr/lib64/nagios/plugins/check_sensors
# general design http://www.kernel-panic.it/openbsd/nagios/nagios6.html
# case -w http://www.linuxquestions.org/questions/programming-9/ash-test-is-string-a-contained-in-string-b-671773/
# Improve:
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
PROGNAME=`basename $0`
PROGPATH=`echo $0 | sed -e 's,[\\/][^\\/][^\\/]*$,,'`
REVISION="0.0.1"
. $PROGPATH/utils.sh
print_usage() {
cat <<EOF
Usage: $PROGNAME -w <thresh_warn> -c <thresh_crit>
EOF
}
print_help() {
print_revision $PROGNAME $REVISION
echo ""
print_usage
echo ""
echo "This plugin checks for the number of active apache threads."
echo ""
support
exit $STATE_OK
}
# MAIN
# Total httpd threads
tot_apache_threads="$( ps -ef | grep -ciE "httpd$" )"
verbosity=0
thresh_warn=
thresh_crit=
while test -n "${1}";
do
case "$1" in
--help|-h)
print_help
exit $STATE_OK
;;
--version|-V)
print_revision $PROGNAME $REVISION
exit $STATE_OK
;;
-v | --verbose)
verbosity=$(( verbosity + 1 ))
shift
;;
-w | --warning | -c | --critical)
if [[ -z "$2" || "$2" = -* ]];
then
# Threshold not provided
echo "$PROGNAME: Option '$1' requires an argument."
print_usage
exit $STATE_UNKNOWN
elif [[ "$2" = +([0-9]) ]];
then
# Threshold is a number
thresh="$2"
# use for a percentage template, from reference 2
#elif [[ "$2" = +([0-9])% ]]; then
# # Threshold is a percentage
# thresh=$(( tot_mem * ${2%\%} / 100 ))
else
# Threshold is not a number or other valid input
echo "$PROGNAME: Threshold must be an integer."
print_usage
exit $STATE_UNKNOWN
fi
case "$1" in *-w*) thresh_warn=$thresh;; *) thresh_crit=$thresh;; esac
shift 2
;;
-?)
print_usage
exit $STATE_OK
;;
*)
echo "$PROGNAME: Invalid option '$1'"
print_usage
exit $STATE_UNKNOWN
;;
esac
done
if test -z "$thresh_warn" || test -z "$thresh_crit";
then
# One or both values were unspecified
echo "$PROGNAME: Threshold not set"
print_usage
exit $STATE_UNKNOWN
elif test "$thresh_crit" -le "$thresh_warn";
then
echo "$PROGNAME: Critical value must be greater than warning value."
print_usage
exit $STATE_UNKNOWN
fi
if test "$verbosity" -ge 2;
then
# Print debugging information
/bin/cat <<EOF
Debugging information:
Warning threshold: $thresh_warn
Critical threshold: $thresh_crit
Verbosity level: $verbosity
Apache threads: ${tot_apache_threads}
EOF
fi
if test "${tot_apache_threads}" -gt "${thresh_crit}";
then
# too many apache threads
echo "APACHE CRITICAL - $tot_apache_threads"
exit $STATE_CRITICAL
elif test "${tot_apache_threads}" -gt "${thresh_warn}";
then
echo "APACHE WARNING - $tot_apache_threads"
exit $STATE_WARNING
else
# fine
echo "APACHE OK - $tot_apache_threads"
exit $STATE_OK
fi

Walking through the code

I included the code above so it gets cached by web crawlers. You should look at the code on github so you get the proper indentations, and line numbers.

So the general format of this script I got from a local file, check_sensor, and Reference 1 below.

The utils.sh call provides nagios-related definitions, including the exit codes that you see used like $STATE_OK.

The shell script is pretty self-explanatory, really. The variables are initialized and the actual checked value is calculated (ps -ef | grep httpd). About half the script (lines 51-100) is parsing the parameters, which is a nice, simple solution if you have predictable and simplified input (like from nagios) and you don’t do the proper parameter parsing that includes -XvalueofXhere with no space between the flag and the value.

Some sanity checking for threshholds (102-113) and debugging information if given enough verbosity (115-125), and then the actual results are determined in 127-140.

Final thoughts

The hardest part of using this plugin is not writing, using, or deploying the shell script. The hardest part is getting the script to run. To use this check properly, you actually need to write a nagios checkcommand like so:
$USER1$/check_by_ssh -H $HOSTADDRESS$ -C "$USER1$/check_apache_threads -w $ARG1$ -c $ARG2$"
With the arguments as the numbers for your thresholds. I used the values 50 and 150 for warning and critical.

Any questions?

References

Weblinks

  1. General design http://www.kernel-panic.it/openbsd/nagios/nagios6.html
  2. case -w http://www.linuxquestions.org/questions/programming-9/ash-test-is-string-a-contained-in-string-b-671773/

Adding ADFS integration to Apache

Overview

ADFS is Microsoft Active Directory Federated Services. It is a single sign-on solution, and this post explains how to tie in Apache 2.4 (CentOS 7) to ADFS. All of this works even with SELinux enforcing!

The test environment described by this document includes the following.

Root URL: https://sample.example.org (10.1.9.192)

Protected subdirectory: https://sample.example.org/auth1/

Warning! ADFS Configuration changes can take a few minutes to take effect.

Adding ADFS integration to Apache

This guide assumes you have a functional apache environment.

Configuring apache

Install mod_auth_mellon from the regular centos repository. Also include php.

yum -y install mod_auth_mellon php

Set up mellon with the sample hostname and url using the provided tool.

mkdir -p /etc/httpd/mellon
cd /etc/httpd/mellon
/usr/libexec/mod_auth_mellon/mellon_create_metadata.sh urn:samplesite:sample.example.org "https://sample.example.org/auth1/endpoint/"

This script outputs 3 files to the current directory.

urn_samplesite_sample.example.org.key
urn_samplesite_sample.example.org.cert
urn_samplesite_sample.example.org.xml

This certificate is a self-signed certificate, but other options can be used and should be considered for production environments. Be aware that the certificates are also dumped into the xml file that will be shared with the ADFS host, so be sure to share any new certificates there as well.

Collect the ADFS metadata and store it locally. Such metadata is usually available at a URL similar to the following.

https://adfs.example.org/federationmetadata/2007-06/FederationMetadata.xml

Here is a copy-pastable line for people like me.

wget https://adfs.example.org/federationmetadata/2007-06/FederationMetadata.xml -O /etc/httpd/mellon/FederationMetadata.xml

Build the apache Mellon config.

cat <<EOF >/etc/httpd/conf.d/auth_mellon.conf
MellonCacheSize 100
MellonLockFile /var/run/mod_auth_mellon.lock
MellonPostTTL 900
MellonPostSize 1073741824
MellonPostCount 100
MellonPostDirectory "/var/cache/mod_auth_mellon_postdata"
EOF

My test environment uses a modular ssl directives include file:

cat <<EOFSSL >/etc/httpd/sites/all-ssl.cnf
SSLEngine on
SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder on
SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"

<Files ~ "\.(cgi|shtml|phtml|php3?)$">
        SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
        SSLOptions +StdEnvVars
</Directory>

SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

SetEnvIf User-Agent ".*MSIE 4\.0b2.*"                 nokeepalive ssl-unclean-shutdown                 downgrade-1.0 force-response-1.0

LogLevel warn
ErrorLog logs/ssl_error_log
CustomLog logs/ssl_access_log combinedvhost

<Directory "/var/www/html/notfound/">
        AllowOverride None
        Order allow,deny
        Allow from all
</Directory>

# END OF FILE all-ssl.cnf
EOFSSL

In the apache config, probably /etc/httpd/sites/sample.conf, modify the virtual host.

cat <<EOF >/etc/httpd/sites/sample.conf
Listen 10.1.9.192:80
Listen 10.1.9.192:443

<VirtualHost 10.1.9.192:80>

        ServerName      sample.example.org:80
        ServerAlias     sample

        # Redirect everything to the https site
        RewriteEngine   On
        RewriteRule ^(.*)$      https://%{HTTP_HOST}%{REQUEST_URI}

</VirtualHost>

<VirtualHost 10.1.9.192:443>

        ServerName      sample.example.org:443
        ServerAlias     sample sample.example.org
        DocumentRoot /var/www/html/sample.example.org

        Include sites/all-ssl.cnf

        <Directory "/var/www/html/sample.example.org">
            AllowOverride None
            Order allow,deny
            Allow from all
            Options Indexes FollowSymLinks
        </Directory>

        <Location /auth1/>
                # Mellon auth which goes to ADFS
                Include sites/adfs.cnf
                MellonCond "groups" "WebAppUsers_grp" [REG,SUB,NC]
        </Location>
</VirtualHost>
EOF

Make the post dump location, which is not necessary but might be useful in the future.

mkdir -p /var/cache/mod_auth_mellon_postdata
chown apache:apache /var/cache/mod_auth_mellon_postdata
chmod 0700 /var/cache/mod_auth_mellon_postdata

Building the include files

We already built the all-ssl.cnf include file, so we just need the adfs include file.

cat <<EOF >/etc/httpd/sites/adfs.cnf
# File: /etc/httpd/sites/adfs.cnf
MellonEnable "auth"
Require valid-user
AuthType "Mellon"
MellonVariable "cookie"
#MellonSamlResponseDump On

MellonSPPrivateKeyFile /etc/httpd/mellon/urn_samplesite_sample.example.org.key
MellonSPCertFile /etc/httpd/mellon/urn_samplesite_sample.example.org.cert
MellonSPMetadataFile /etc/httpd/mellon/urn_samplesite_sample.example.org.xml
MellonIdPMetadataFile /etc/httpd/mellon/FederationMetadata.xml
MellonMergeEnvVars On ":"
MellonEndpointPath /auth1/endpoint
EOF

Here, the MellonMergeEnvVars On “:” means that any multiple-value attribute (like Groups) will be added to one colon-delimited string instead of being assigned to “Mellon_Groups_1” “Mellon_Groups_2” and so on. It’s how the MellonCond works in the virtual host configuration.

Building example sites

Build the index file for the protected directory.

cat <<EOF >/var/www/html/sample.example.org/auth1/index.html
<html>
<head><title>Authorized zone</title></head>
<body>
<h1>Welcome to the authorized zone.</h1>
You should only be able to see this if you are authenticated and authorized.
</body>
</html>
EOF

Make a php troubleshooting file.

thisfile=/var/www/html/sample.example.org/auth1/info.php
cat <<EOF >${thisfile}
<?php    
phpinfo(INFO_VARIABLES);
?>
EOF
chown apache:apache ${thisfile}
chmod 644 ${thisfile}

This little php file will show the apache environment variables that are available for use in the apache directives. The important ones here will be the ones prepended with “MELLON_.”

Configuring ADFS to share data

On the ADFS server, add a new relying party trust.

Run the AD FS management tool.

Navigate in the tree structure to AD FS –> Trust relationships –> Relying party trusts.

Select on the action menu “Add relying party trust…”

The easiest way to do this is to use the xml file generated by that script earlier.

Do not configure multi-factor authentication.

Permit all users to access this relying party.

Edit the properties of the relying party trust –> Advanced tab.

Set value “Secure hash algorithm” to SHA-1.

Adding claim rules

Right-click this relying party trust and select “Edit Claim Rules.”

Add a rule of type “Transform incoming claim.”

Incoming claim type: Windows account name
Outgoing claim type: Name ID
Outgoing name ID format: Transient Identifier
Radio button: Pass through all claim values

The rule text looks like:

c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname"]
 => issue(Type = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier", Issuer = c.Issuer, OriginalIssuer = c.OriginalIssuer, Value = c.Value, ValueType = c.ValueType, Properties["http://schemas.xmlsoap.org/ws/2005/05/identity/claimproperties/format"] = "urn:oasis:names:tc:SAML:2.0:nameid-format:transient");

Reference: Specific example of mellon configuration

Add custom rule “Get all groups user belongs to.”

c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"]
 => add(store = "Active Directory", types = ("http://schemas.xmlsoap.org/claims/Group"), query = ";tokenGroups;{0}", param = c.Value);

The add command instead of issue passes the information on to the next rule.

Reference: Custom rules sharing

Add custom rule “Filter your groups only.”

c:[Type == "http://schemas.xmlsoap.org/claims/Group", Value =~ ".*WebAppUsers_grp.*|.*IT.*"]
 => issue(Type = "groups", Value = c.Value, Issuer = c.Issuer);

The type name of “groups” means that mellon will provide an apache environment variable of “MELLON_groups” which we are performing the condition statement on in the virtual host.

The regex in the first part is looking for any group name that has “IT” in it at all or “WebAppUsers_grp” which is probably pretty specific to just that one group.

What this does is limit the groups being sent so instead of those 538 AD groups possible that that one user is in, it will pass back only the 28 IT department-related ones and the WebAppUsers_grp.

After the apachectl configtest, give apache graceful a shot!

https://sample.example.org/auth1/info.php

The above link (modified for you of course) should redirect to the AD FS login page and then send authenticated users back!

The benefits here include using apache as a reverse proxy to tomcat applications (local or otherwise) and providing a layer of authentication.

References

  1. Official mellon page https://github.com/UNINETT/mod_auth_mellon
  2. Specific example of mellon configuration https://answers.splunk.com/answers/177936/accessing-splunk-enterprise-using-adfs-authenticat.html
  3. https://httpd.apache.org/docs/2.4/mod/mod_authz_core.html
  4. ADFS saml rules https://support.zendesk.com/hc/en-us/articles/203663896-Mapping-attributes-from-Active-Directory-with-ADFS-and-SAML-Professional-and-Enterprise-
  5. Regex is allowed in claims https://social.technet.microsoft.com/wiki/contents/articles/8008.ad-fs-2-0-selectively-send-group-membership-s-as-a-claim.aspx
  6. http://serverfault.com/questions/700126/sending-ad-attributes-as-ad-fs-claims-to-shibboleth-sp-attributes
  7. Custom rules sharing http://molikop.com/2014/04/adfs-claim-rules-filtering-groups/
  8. http://stackoverflow.com/questions/6861534/is-it-possible-to-output-any-or-all-available-variables-in-a-htaccess-file