README/FAQ for logfilterd, written by Adam Spiers <adam@spiers.net>
==============================================================================

Introduction and motivation
---------------------------

This program is a log filtering "daemon" written in Perl.  I decided to
write it after evaluating several (all?) the log filtering programs
currently available (swatch, logsurfer, logcheck etc.) and finding that
to my mind, none of them provided the full flexibility I needed.  I
wanted something that would:

  - eliminate uninteresting verbosity in /var/log/messages, thereby
    making it very easy to spot anything out of the normal (which I
    consider a Very Good Thing for a sysadmin to be able to do).
    Simply tweaking syslogd.conf is an unacceptable solution for many
    reasons, but not least because the policies adopted by different
    software packages for what priority levels they use when logging
    with syslog vastly differ.

  - siphon off different types of log data into different files
    (e.g. one log file for all potential hacking attempts, one for all
    ftpd transactions etc.)

  - rapidly alert me in certain situations (e.g. detection of port scans
    and other attacks)

  - provide me with concise summaries of certain events, even if they
    occurred continuously over several minutes or even hours

  - be pretty quick

  - not refilter whole input log files if you altered the filtering
    algorithm (unless you really wanted it to)

  - have the power of Perl regexps and string handling in its full glory
    :-)

I decided to make the daemon a single script rather than an immutable
code base together with a complex configuration file, and in doing so
obviously gained pretty much unlimited flexibility in how I wanted my
logs filtered.  Of course, the downside was that anyone else who wanted
to use the code would need enough programming knowledge to be able to
read Perl code and alter it in simple ways.  However, I deliberately
wrote the code so that this would pose very little difficulty for the
average programmer/Linux enthusiast.

I wrote it specifically for the machine thelonious.new.ox.ac.uk, which
is a RedHat Linux box running a slightly unusual combination of
services; hence it will need to be adapted to your machine before you
can use it.  However, the structure of the program should be very clear,
especially if viewed in emacs with folding mode (folding.el) loaded.
(Wow, never thought I hear myself come out with a 'best viewed in ...'
line.)


What does it actually do?
-------------------------

In addition to filtering out a lot of common log messages of little
interest, it splits the output into several different files (one for
each daemon), sets up 'contexts' a la logsurfer (see
http://www.cert.dfn.de/eng/team/wl/logsurf/ for information about
'contexts' to help you understand more precisely what this is all about)
in order to compress noisy security events (e.g. a big portscan) into a
readable summary, and sends e-mail alerts such as the one above
immediately upon detection of something more serious like a portscan.
The current version has whittled down a 10MB /var/log/messages to a 1MB
/var/log/filtered-messages, which makes for additions which can be
easily read every day without consuming too much time.  The patterns it
matches are found towards the end of the code.

Here's a sample e-mail it might (and in fact did) generate.

-------- 8< -------- 8< --------
Date: 20 Jul 1998 06:59:03 -0000
From: anonymous@thelonious.new.ox.ac.uk
To: adam@thelonious.new.ox.ac.uk
Subject: logfilterd ALERT: port scan detection

Port scan (context id #3) detected from host:
   pigpen.pcl.ox.ac.uk/163.1.218.50,

starting at Jul 20 05:20:00, and 
ending   at Jul 20 05:20:00.

SYN/Normal scan employed on TCP ports: 53
-------- 8< -------- 8< --------

And here are some of the lines which might appear in your
/var/log/filtered-messages following port scans.  The number in the []
parentheses is the context number (these numbers are allocated
sequentially, but reclaimed as the contexts finish).

-------- 8< -------- 8< --------
May 31 19:11:18 logfilterd[sentry/3]: Detected port scan initiating from DIALIN0165.UPENN.EDU/128.91.16.165
May 31 20:15:54 logfilterd[sentry/3]: UDP scan employed on UDP ports: 137
May 31 20:15:54 logfilterd[sentry/3]: Port scan from DIALIN0165.UPENN.EDU/128.91.16.165 finished at May 31 19:11:46
Jun  1 07:24:08 logfilterd[sentry/4]: Detected port scan initiating from james.eimc.brad.ac.uk/143.53.203.241
Jun  1 11:07:58 logfilterd[sentry/4]: SYN/Normal scan employed on TCP ports: 1--20, 24, 26--78, 81--109, 112, 114--122, 124--142, 145--176, 178--194
Jun  1 11:07:58 logfilterd[sentry/4]: Port scan from james.eimc.brad.ac.uk/143.53.203.241 finished at Jun  1 07:24:52
Jun  1 11:07:58 logfilterd[sentry/5]: Detected port scan initiating from colorgfx.demon.co.uk/158.152.162.35
Jun  1 13:10:15 logfilterd[sentry/5]: UDP scan employed on UDP ports: 137
Jun  1 13:10:15 logfilterd[sentry/5]: Port scan from colorgfx.demon.co.uk/158.152.162.35 finished at Jun  1 11:08:01
Jun  1 13:10:15 logfilterd[sentry/6]: Detected port scan initiating from hail.pipex.net/158.43.128.104
Jun  1 15:10:17 logfilterd[sentry/6]: UDP scan employed on UDP ports: 7
Jun  1 15:10:17 logfilterd[sentry/6]: Port scan from hail.pipex.net/158.43.128.104 finished at Jun  1 13:11:21
-------- 8< -------- 8< --------


How do I install it?
--------------------

See INSTALL for rough installation instructions.


I've tried installing it but I'm stuck!  What now?
--------------------------------------------------

You can try e-mailing me <adam@spiers.net> with a very detailed and
precise description of the problem, and I may help, but I spent very
little time computing these days and rarely check my e-mail anyway, so
you might be better off asking elsewhere ...

Needless to say, any questions which show no signs of consulting Perl
man pages and/or FAQs where appropriate will be redirected straight to
/dev/null.


Gee, that's useful, thanks.  But how about changing it like this?
-----------------------------------------------------------------

If you've extended/changed it in a way which may be useful to others,
wonderful --- let me know (long live the Bazaar!).  This applies to any
suggestions you might have too, even if it's about the slightly muddled
documentation.  Ideally I'd like someone to take it over and develop it
properly.  Building an .rpm would also be an easy and worthwhile job.
Any offers?
