log2db - Log-file to Berkeley database converter (c) Andreas Gruenbacher , 2001-2002 Last update to this document: 20 April 2002 Primary location: http://www.bestbits.at/log2db/ ==================== Update Notifications ==================== If you want to get notified when new versions of log2db become available, please send a message to log2db@bestbits.at. If you want to subscribe under an address different from your `From:' address, please specify which address I should use. =============================== Installation and Use Guidelines =============================== Log2db is a simple tool that watches a log-file and scans for specific entries (those entries are specified with a regular expression). It inserts the entries found into a Berkeley database for a specified amount of time, after which the entries are deleted from the database again. This functionality is exactly what is needed for a POP-before-SMTP solution on a host which is running sendmail and a pop daemon. probably log2db also works for other daemons with similar requirements. Log2db includes the following features: * Watching a log-file and updating the database. * Support for Berkeley hash and btree databases (db versions 1, 2, 3, 4). * Database locking compatible with sendmail (using flock(2)). * Automatic recognition of log-file rotation and truncation. * Manual monitoring and maintenance of the database. * A configurable amount of information is send to the system log. The rest of this document describes the steps needed to implement a POP-before-SMTP solution using qpopper, this tool, and sendmail. LICENSE ======= Log2db is licensed under the GNU General Public License. See the LICENSE file for details. REQUIREMENTS ============ In principle log2db should run on UNIX like systems that have a version of Berkeley DB. It has been tested in SuSE systems; people have reportedly succeeded in running log2db on RedHat systems as well. If used for POP-before-SMTP, the pop daemon must be configured to log successful login events to a file, possible via the system log. (This may require rebuilding or reconfiguring the pop daemon.) Instead of POP, an IMAP or similar daemon should work equally well. On a SuSE 7.1 system with Qualcomm Qpopper 3.1.2, the qpopper package needs to be rebuilt with the additional configuration option --enable-log-login. Sendmail must have been built with Berkeley database support (which is usually the case). Berkeley DB versions 1, 2, 3, and 4 are supported. CONFIGURING LOG2DB ================== Since version 0.5, logd2b uses GNU autoconf, and thus is configured with: $ ./configure Configure tries to guess the configuration. There are situations in when this is not possible entirely unaided. Therefore, configure has a number of parameters (see `configure --help'). Configure by default tries to use the Berkeley DB header file db.h in the standard include path, and tries to link against the libdb.so or libdb.a library in the library search path. If the db.h file that should be used is not in the standard include path, use the the CPPFLAGS parameter of configure to specify the directory the header file is located in as an additional include directory: CPPFLAGS=-I If the Berkeley DB library that should be used is not located in the library search path, use the LDFLAGS parameter to specify the directory the library file is located in as an additional library directory: LDFLAGS=-L Finally on some systems, multiple versions of Berkeley DB are installed, each of the library files called differently. If log2db should be linked against a Berkeley DB library other then libdb.{so,a}, specify the library base name (e.g., `db', `db-3', `db2', `db1') using the --with-db parameter: --with-db= Please make sure you are using a Berkeley DB header file that matches the Berkeley DB library file. If you fail to do so, the link step of building log2db will fail. Please read the output of `configure' carefully. It may contain warnings or error messages. If `configure' is not able to guess a configuration by itself, it will not generate the necessary output files which `make' requires. As an example, on my system the following `configure' options will build versions of log2db using Berkeley DB versions 1, 2, 3, and 4: VERSION CPPFLAGS LDFLAGS --with-db 4 (none) (none) (none) 4 -I/usr/include/db4 (none) db or db-4 3 -I/usr/include/db3 (none) db or db-3 2 -I/usr/include/db2 (none) db2 1 -I/usr/include/db1 (none) db1 BUILDING AND INSTALLING LOG2DB ============================== After configuring, build log2db with: $ make clean $ make After log2db has built correctly, verify that the version of Berkeley DB log2db is using matches the version sendmail is using. To do that, (1) Build a database using makemap (which is part of sendmail). The makemap utility is located in /usr/sbin/ on my system. You may have to specify the full path name. Example command: $ date "+1.2.3.4 %s" | /usr/sbin/makemap hash test.db (2) Check the file Makefile. Adjust the DB variable to match the Berkeley DB version sendmail uses. (3) Check whether the resulting log2db binary can read the database: $ log2db --db=test.db --dump -000:10 1.2.3.4 If log2db fails to print 1.2.3.4, and prints out error messages like this: $ log2db --db=test.db --dump log2db: test.db: Invalid argument or this: $ log2db --db=test.db --dump log2db: unrecognized file type then the version of Berkeley DB log2db is using is different from the version Sendmail is using. In that case, try another Berkeley DB version (see CONFIGURING LOG2DB above). Note that log2db has been tested on Linux; on other systems, several modifications may be required to get the utility to run. INSTALLATION ============ You can install log2db with: $ make install Log2db and the log2db-wrapper script are installed in /usr/local/sbin by default. You can specify a different directory with the `--sbindir' parameter to `configure'. Next, check to which file your POP daemon logs successful login events. Depending on your syslog configuration, these messages may end up in /var/log/mail, /var/log/messages, or somewhere else. In my example, messages are written to /var/log/mail. Since I am using Qpopper, the log entries look like this: ... POP login by user "username" at (some.host.name) 1.2.3.4 ... The University of Washington POP/IMAP daemons produce log entries like this: ... Login user=username host=some.host.name [1.2.3.4] ... There are also other log entry formats. Review the settings in the log2db-wrapper script, and adjust them accordingly. The regular expressions for Qualcomm Qpopper and the University of Washington IMAP/POP daemons are already included in log2db-wrapper. The script itself contains some documentation about the settings. I recommend you test whether log2db correctly recognizes your POP daemon's log file entries. If it does not, you will need to adjust the regular expression. See `Debug mode' at the end of this document. You can force `log2db-wrapper' into debug mode by commenting out the DBFILE variable. Information about running and testing log2db can be found after the CONFIGURING SENDMAIL section. In order to get log2db to start when your system starts up, you need to add it to your init script(s). On a system with System V style init scripts, you may find the file SuSE/log2db-7.3 useful. It is taken from a SuSE 7.3 system. After copying this script to the startup scripts directory (usually, /etc/rc.d/), the insserv utility can be used to add the script to the appropriate run levels. On a SuSE 7.1 system, in order for log2db to start at system start time, you will also need to add the following option to /etc/rc.config: START_LOG2DB=yes The file RedHat/popB4smtp should work for a RedHat like system. Thanks go to Kevin L. Lucier for providing this script. CONFIGURING SENDMAIL ==================== On www.sendmail.org an m4 script that implements HACK(`popauth') can be found at . This file is also included in this package. If you want to use this script, copy the file popauth.m4 into the directory that holds your sendmail hack options (on my system, this is /usr/share/sendmail/hack/). Then, add the following two options to the m4 script used to generate your mail sendmail configuration file (usually, /etc/sendmail.cf). (On my system, this is /etc/mail/linux.mc). define(`POP_B4_SMTP_TAG', `') HACK(`popauth') The file /etc/sendmail.cf can then be regenerated with the following command: m4 /etc/mail/linux.mc > /etc/sendmail.cf The hosts which are allowed access via the popauth mechanism are authorized in addition to the existing configuration, i.e., it is not necessary to add hosts that were allowed to relay via the local sendmail to the popauth database. Other configuration issues for roaming users are discussed at http://www.sendmail.org/~ca/email/roaming.html. RUNNING LOG2DB ============== Running log2db as a daemon -------------------------- Invoke /usr/sbin/log2db-wrapper as a background process (&) to start log2db. The wrapper script passes the necessary parameters to log2db, and restarts it in case it should terminate unexpectedly. You can terminate log2db by sending the wrapper script a kill signal. The process identifier (pid) of the wrapper script can be found in the file /var/run/log2db.pid. Manual database maintenance --------------------------- You can also use log2db to manually create, dump, and expire the database, add and delete records. In order to do that, invoke log2db with the database name, and perhaps the database type (log2db assumes hash as the default type). After that, specify one of the operations --create, --dump, --expire, --add IP, --del IP. Examples are: log2db --db=popauth.db --create log2db --db=popauth.db --add 1.2.3.4 log2db --db=popauth.db --dump Debug mode ---------- Usually the last argument to log2db is the name of the log-file to watch. If this argument is omitted, log2db enters into a debug mode in which lines are read from standard input, and compared with the regular expression. Log2db only tries to match the lines passed to it against the regular expression; no database operations will occur. This can be used to try out different regular expressions. Example: log2db --regex='login ([.0-9]*)' log2db --regex='Login user=(.*) host=([^ ]*) ?\[([.0-9]*)\]' \ --regex-key='\3' --regex-value='\1@\2' For each matching log line, log2db prints out the key and value: 1.2.3.4 (username@some.host.name)