Abstract

This document describes a proposed log message hub ("hub") and user interface ("UI") to complement the logging system described in [PEP282] and implemented in the python logging system.

The hub and UI would perform the following duties:

• accept messages from the logging module via the logging module's handlers (FileHandler, SocketHandler, SyslogHandler, SMTPHandler, NTEventLogHandler, SQLHandler)

• redistribution of messages to other targets (just plucking ideas from the ether here) ... all the above Handlers, ElvinHandler, OpenviewHandler, SunViewManagerHandler, JabberHandler, SMSHandler

• perform some action on incoming messages, possibly aggregating, dropping or forwarding

• have state which would be affected by the actions and may cause new messages to be generated (ie. escalation)

• present a display of the messages to a user, either graphically or textually (the most basic of these would resemble the UNIX tail -f)

Note: it is not the intention of this proposal to implement a monitoring system like EDDIE. Monitoring of systems is performed by other components which generate log messages. It is the intention of this proposal to present those messages to users in meaningful and useful ways.

Note: it is not the intention of this proposal to add this functionality to the core logging implementation of PEP 282. The implementation of this proposal would form a separate installation of software.

Message Handling

The python logging system implements the following structure:

in -> Manager -> Logger -> Handlers -> Formatter -> out

The "out" side feeds into the hub. The hub is also capable of feeding messages back into itself. The hub would be implemented as the following structure:

               External Message Generation
                          |
                          V
          +----------+         +----------+
          | Receiver | .. N .. | Receiver |
          +----------+         +----------+
                          |
                          V
+-------------+    +-----------+         +-------+
| Config Port |--->|           |<--------| Timer |
+-------------+    |    Hub    |         +-------+
       +-----------|           |<-------+
       |           +-----------+        |
       |                  |         +-----------------+
       |                  |         | LoopbackHandler |
       V                  V         +-----------------+
   +--------+         +--------+        |
   | Action | .. N .. | Action |--------+ new LogRecord
   +--------+         +--------+
       | new LogRecord |      ^
       |               |      |
       V               V      v
  +---------+   +---------+  +-------+
  | Handler |   | Handler |  | Store |
  +---------+   +---------+  +-------+
       |
       V
  +--------+
  | Client |
  +--------+

The LogRecord and Handler implementations are used from the PEP 282 implementation. Additional Handlers will be implemented as required. Filters are available at the Receivers, Hub, Actions and Handlers.

Configuration

The configuration of the hub will ultimately determine whether the messages are:

• forwarded on to other logging mechanisms

• forwarded on to clients

• filtered for manipulation or discarding

• stored

The initial configuration will describe:

Receivers

The receivers active in the hub and what they're listening to.

Actions

The actions that are set up by default.

Clients

The client configuration - who is allowed to connect and how. Some sort of access control spec.

At the moment, it's looking like the configuration will be done in a python module that actually launches the service/daemon.

Simple Configuration (minus imports):

hub = logext.Hub()

h = logging.StreamHandler()
f = logging.Formatter("%(asctime)s %(name)-19s - %(message)s")
h.setFormatter(f)
f = logging.Filter()
f.level = ['ERROR']
a = logext.Action()
a.addHandler(h)
a.addFilter(f)
hub.addAction(a)

def tcpReceiver():
    r = logext.TCPReceiver(hub, 'localhost', logging.DEFAULT_TCP_LOGGING_PORT)
    print 'tcp started'
    r.serve_forever()

def udpReceiver():
    r = logext.UDPReceiver(hub, 'localhost', logging.DEFAULT_UDP_LOGGING_PORT)
    print 'udp started'
    r.serve_forever()

The receivers are now started in threads (see test_hub.py).

The basic Receiver, Hub, Action, Filter and Handler objects will be configurable using a simple XML configuration file:

<hub>
 <receiver type="TCP">
   <port> DEFAULT </port>
   <hostname> localhost </hostname>
 </receiver>

 <receiver type="TCP">
   <port> DEFAULT </port>
   <hostname> remotehost </hostname>
 </receiver>

 <action>
  <filter>
   <level> CRITICAL </level>
  </filter>
  <handler type="File">
   <filename> all_critical.log </filename>
  </handler>
 </action>
</hub>

Additionally the client CONFIG messages may include remove commands:

<hub>
 <remove>
  <action id="[id]"/>
  <handler id="[id]"/>
 </remove>
</hub>

Replace the nominated Handler with this one:

<hub>
 <handler type="File" id="[id]">
  <filename> alternative.log </filename>
 </handler>
</hub>

Set up a new client connection with all INFO messages:

<hub>
 <action>
  <filter>
   <level> INFO </level>
  </filter>
  <handler type="Client"/>
 </action>
</hub>

Use Case Scenarios

GUI monitoring of log messages

Several systems generate logging messages which are collected in a central place. Operations personnel have a monitor client on their desktop which displays messages pertinent to them.

The events for a GUI connection look like:

client config message

Message comes in through the Config Port with information describing the message filtering that the client wishes to set up.

hub creates new Action

New Action is created with appropriate filters

client connected to Action

Hub passes connection from config port to action's handler. This part is a little (ok, a lot ;) hazy.

SMS alerting of critical messages

A series of daemon restarts in succession alerts on-call operators via SMS that something very bad is happening.

Remote site monitoring

A hub at the remote site filters the messages. It passes the filtered messages on to a hub at the local site for operations staff to monitor. The remote site hub may even take remedial action based on log messages.

Clients cancelling escallations and silencing messages for a period

Clients will want to be able to indicate to the Hub that a message has been dealt with, and that it should not escallate. Also, they will want to be able to squash messages for a period of time (expected downtime, or "retroactively scheduled downtime .... I'm trying to fix the %@#$% thing will someone shut the @#$%@!#$ escallation system up")

The events for escallation cancellation look like:

initial message

message comes in and an Action recognises it as something that should be actioned within 5 minutes or an escallation should be generated. It stores the time of the message off in a state file.

cancellation

a client indicates to the Action that the message should be

escallation

five minutes pass (300 ticks of the Hub's timer) and nothing has happened, so the Action fires an escallation message off to its Handler (possibly email, possibly SMS, possibly voicemail playback)

The events for escallation squash are:

initial message

as above

squash message

CONFIG message comes in on the Hub's dedicated config port and is intercepted by the Hub. The message is decoded to extract the squash filter parameters and duration. A squash filter is then set up which destroys all messages of the given type until the duration passes. The duration is specified by the end filter parameter, and once it's passed the Hub is able to remove the Filter.

Others???

Implementation

See logext.py for an initial implementation that provides the Hub, Action and Receiver base classes. It also implements TCP and UDP receivers.

It uses a modified logging.py and logrecv.py.

Test it with the test_hub.py server. Send messages to the server with log_test3.py (which uses logging.cfg).

TODO: the default Filter described here could probably be made the default filter in the logging.py module...

TODO: does the Filter start/end refer to time.time() or record time?

TODO: we need to be able to ID Filters and Actions to configure them on the fly!

References

[PEP282]

http://python.sourceforge.net/peps/pep-0282.html: A Logging System

Python Logging System: http://www.red-dove.com/python_logging.html

EDDIE: http://eddie-tool.net/

Thanks

Thanks to Trent Mick, Vinay Sajip, Anthony Baxter and Dave Cole for their inspiration, code hackery and feedback on this proposal.

Copyright

This document has been placed in the public domain.