Introduction
------------------------------------------------------------
dtquery is a CGI-based tool to query your mon downtime logs for
specific downtime events, on specific hosts/groups/services, during
specified date ranges, and to supply you with graphs summarizing the
results. 

Downtime can also be queried on a per-host basis, even though mon
doesn't support the feature officially. When most services fail, the
monitor which detected the failure writes the names of the failed
hosts into the summary line, and the summary field is recorded as part
of the downtime log. When we are searching for "hosts", what we are
actually doing is searching for "text in the summary field", but in
most configurations, these are identical.

dtquery was developed so that we could analyze our downtime records more
effectively, and more easily answer questions like:
   1) When are certain types of failures typically occurring? Are
   there time of day/week/month patterns?
   2) Are certain hosts within hostgroups more vulnerable to failures
   than others?
   3) Are certain services within hostgroups more vulnerable to
   failures than others?
   4) When failures happen, how long do they last and what does the
   distribution of failure times look like?
   5) Why should we use mon and not replace it with another
   open-source or commercial monitoring package that has more
   graphing/reporting features?

dtquery was developed and tested on Solaris 7 (sparc). It should work
on any UNIX that supports the underlying software (mon, perl, gd,
gnuplot), basically, any system that can run mon and mon.cgi should
also be capable of running dtquery.


Installation Instructions
------------------------------------------------------------
1. You must have a working mon installation that is generating
   downtime logs. See the mon documentation on how to specify this if
   you haven't already (you must specify 'dtlogging = yes' and
   'dtlogfile = /path/to/dtlogfile' in your mon.cf file). You will
   also need a reasonable amount of downtime data in your logs in
   order for this tool to generate significant value and produce
   meaningful graphs. Download mon at:
      ftp://ftp.kernel.org/pub/software/admin/mon/

2. Although it's not strictly required, you should also install a new
   version of mon.cgi, which is integrated into dtquery. mon.cgi is
   available from the same location where you got mon, and includes
   installation instructions:
      http://www.nam-shub.com/files/

3. Install zlib and libpng on your system, if they are not already
   available. You may need to specify building shared libraries for some
   architectures, we needed to specify building shared libraries for
   libpng. Both png and zlib are available at:
      ftp://ftp.uu.net/graphics/png

4. Install a png-capable version of gd on your system. This includes
   all recent versions of gd, we used v1.8.3 and that is what we
   recommend. The gd graphics library is available from:
	      http://www.boutell.com/gd/

5. Make sure the requisite perl modules are installed. These modules
   are all available from CPAN (http://www.cpan.org). You will need:
      Mon::Client
      Statistics::Descriptive
      GD::Graph (requires GD, we used v1.4)
   The remaining required modules (CGI, Time::Local, Carp) all come
   with a standard perl5 build, but will also be available on CPAN.

6. Make sure gnuplot is installed. gnuplot is available from
   http://www.gnuplot.org/. We used v3.7.1, the latest version
   available at the time of release. Make sure you build gnuplot with
   png support (use the "--with-png" option during configure).

7. Test gnuplot to verify that it is properly installed and can
   output png files properly.
# gnuplot
gnuplot> set output '/tmp/test.png'
gnuplot> set term png
Terminal type set to 'png'
Options are 'small monochrome'
gnuplot> plot sin(x)
gnuplot> exit
   Now view the resulting image in a web browser or image-viewing program
   to verify that the image is generated and that it looks like a sine
   wave.

8. Copy the dtquery.cgi script into your webserver's cgi-bin
   directory. If you are running Apache, DO NOT RUN THIS SCRIPT UNDER
   mod_perl, or else YOU WILL SUFFER SEVERE PERFORMANCE PENALTIES!
   This is because dtquery.cgi forks off external gnuplot processes,
   which under mod_perl, means that you are actually forking off an
   entire httpd process to accomplish each fork.  Please see the
   following URL for more information about why mod_perl is a bad idea
   for dtquery.cgi:
   http://perl.apache.org/guide/performance.html#Forking_and_Executing_Subprocess


9. Edit the header portion of dtquery.cgi to reflect your mon
   configuration and your organization's defaults. If you're
   impatient, you can probably leave most of the settings alone, as
   they are set to reasonable defaults. Note that by default, dtquery
   is set up to query a live mon server for downtime log data (with
   $main::dtlog_source == "mon") . You may very well wish to separate
   the downtime log information onto a different machine, in this case
   the "files" option would be used for $main::dtlog_source.



Usage Instructions
------------------------------------------------------------
1. You will need a browser that supports Javascript. Netscape 4 and
   IE5 were both tested and will work.

2. Open up the dtquery web page in your browser. The page may take a
   few moments to load, since it actually makes a query to the mon
   server that you specified and retrieves all current groups,
   services, and hosts. Select your query criteria and go! From there,
   hopefully everything should be obvious. If it's not, then we did
   something wrong. Let us know how we can improve!



What You Can Learn About Your Downtime From The Graphs
------------------------------------------------------------
You can learn a lot about your downtime from the graphs generated. You
can also learn nothing, there's no guarantees that trends will be
apparent or that they will be apparent (or actual trends, and not just
coincidences). Asking the right questions of your data is not always
an easy task, and sometimes, there's just no clear answers. Trends
will not always pop out at you. To help you get more out of the
graphs, here's some hints.

* Downtime by Hour of Day - Also known as "The Bar Code Graph", this
  graph shows a binary representation of the state of the service. Red
  means failure, white means OK. Unless your timeframe is very small
  (1-2 days), or you have very few failures, it may be hard to get
  much out of this graph. But it is very good at showing you date
  ranges to dig deeper within.
* Cumulative Downtime by Time of Day - This graph answers the question
  "what time of day is this host/group/service spending the most
  time in the failure state?"
* Cumulative Downtime by Day of Week - This graph answers the question
  "what day of the week is this host/group/service spending the most
  time in the failure state?"
* Failure Time Distribution - This graph shows you the exact
  distribution of your failure times, in minutes, on a logarithmic
  scale. It answers the questions "Are most of my failures short?
  Long? Is there a discernible pattern?"
* Cumulative Downtime by Service - This graph answers the question
  "For a given group or groups, how is my downtime distributed among
  various services?" For example, how much time has your HTTP service
  failed relative to the ping service?
* Cumulative Downtime by Group - This graph answers the question
  "For a given service or services, how is my downtime distributed among
  different hostgroups?" For example, which are the groups with the
  most minutes spent in the failure state?



Performance
------------------------------------------------------------
dtquery was not designed as a super high-performance application. It
reads in downtime logs, which are flat text files. We have tested the
application in development with large datasets (13000+ event downtime
log) and it performs acceptably on a 333MHz Sparc Ultra-5 for most
queries. The main factor in the running time is the number of results
returned. Searching a 13000-event logfile, getting 600 matches, and
operating on those is not a big deal (2-3 seconds), but returning
13000 events will force dtquery to work hard for a good 30-40
seconds. Probably most of this time is in the sorting.

We haven't done any performance tuning or profiling of the code, so
there are probably significant opportunities for performance
improvements. We felt that for data sets significantly larger than
what we are dealing with now, a database would definitely be the way
to go. 



Ongoing Maintenance
------------------------------------------------------------
One feature of the initial version of dtquery is that it does not
clean up the graphs that it creates . This is so you can make graphs
and then send the display URL to a colleague or post it on a web
page. The current way things are, the URL that would be necessary to
generate a graph would be really long and might not fit in a GET
request, and would certainly be really ugly.

The current way to deal with this is to implement a cron job that
periodically cleans up the graph directory. For example:
  # remove all files in the dtquery graph directory that have not been
  # accessed for 14 days or more.
  0 2 * * * /bin/find /tmp/dtquery-cache -atime +14 -exec /bin/rm {} \;

In the future, we might:
    1) Implement a cleanup job as part of dtquery itself. A small
    overhead in processing for each request, but the cache is kept
    relatively clean.
    2) Stick with implementing cleanup as a cron job.
    3) Shorten the query parameters, and allow the whole query to be
    embedded in the URL in a reasonable way. The tradeoff is disk
    storage vs. CPU utilization to generate the graphs.


Credits
------------------------------------------------------------
The initial version of dtquery, including all Javascript and HTML
coding, the query engine, and the presentation logic, were done by
a colleague who wishes to remain anonymous, without her efforts, dtquery
would not exist in any form. Graphing capabilities were added by
Andrew Ryan (andrewr@mycfo.com). "Code Reuse" was done from Cricket
(png spraying routines), Chart::Graph (gnuplot running routines), mon,
mon.cgi, and our own internal trouble ticketing system.
