Monit - utility for monitoring services on a Unix system

Monit is fairly easy to setup, though may lack features that larger sites may need from a monitoring system. Assuming OpenBSD and some network services and a custom mail filter thing to check on,

    # pkg_add monit
    # rcctl enable monit
    # cat /etc/monitrc
    set daemon  30
    set idfile /var/monit/id
    set terminal batch
    set statefile /var/monit/state
    set mailserver localhost
    set eventqueue
        basedir /var/monit/events
        slots 10
    set mail-format {
    from:    Monit <monit@$HOST>
    subject: monit:$ACTION|$SERVICE|$EVENT
    message: fyi
    }
    set alert FIXME@example.org
    set httpd port 2812 and
        use address localhost
        allow localhost
        allow admin:Hunter2
    check system $HOST
        if loadavg (1min) per core > 2 for 5 cycles then alert
        if loadavg (5min) per core > 1.5 for 10 cycles then alert
        if cpu usage > 95% for 10 cycles then alert
        if memory usage > 75% then alert
        if swap usage > 25% then alert
    check process gmid matching "gmid"
        start program = "/usr/sbin/rcctl start gmid"
        stop program  = "/usr/sbin/rcctl stop gmid"
        if cpu > 60% for 2 cycles then alert
        if cpu > 80% for 5 cycles then restart
        group server
    check process httpd matching "httpd"
        start program = "/usr/sbin/rcctl start httpd"
        stop program  = "/usr/sbin/rcctl stop httpd"
        if cpu > 60% for 2 cycles then alert
        if cpu > 80% for 5 cycles then restart
        group server
    check directory quarantine with path /var/spool/quarantine
        if changed timestamp then alert
    # rcctl start monit
    ...
    # monit status
    ...
    # monit reload
    ...

Monit sends alerts by email, so one should not have a monitor that alerts when an email is sent. Avoid positive feedback loops, in other words. The default message format has been changed to make it somewhat easier for a computer to parse, on the notion that one would email the alerts to a system that would then decide how to or whether to alert, depending on the specific alarm, how many alarms are coming in, the time of day, etc.

Email alerts assume that the mail (and DNS and other) systems are operational, which may not be the case. An alternate means of sending alerts may be necessary, for example to use log scanning and a third-party API to raise an alarm directly to the phone of the on-call. I haven't gotten too deep into monit so do not know what it offers for alternative means of raising alarms.

I am not seeing syslog messages from monit on OpenBSD despite setting "set log syslog" so I'm not sure what's up with that. Logging to a custom logfile would be another option, though do be sure to rotate any such new logfiles, rather than allowing the files to eventually fill up the disk.