Python Logging Message Hub and User Interfaces


Richard Jones




Note some of the components described in this document are implemented in the python logging system. In the course of designing this system, some changes were proposed to the logging implementation. These changes have been accepted but not implemented, so some of the implementation details described here will be out of sync with the reality of the logging system for a while.


This document describes a proposed log message hub ("hub") and user interface ("UI") to complement the logging system described in [PEP282] and implemented in the python logging system.

The hub and UI would perform the following duties:

Note: it is not the intention of this proposal to implement a monitoring system like EDDIE. Monitoring of systems is performed by other components which generate log messages. It is the intention of this proposal to present those messages to users in meaningful and useful ways.

Note: it is not the intention of this proposal to add this functionality to the core logging implementation of PEP 282. The implementation of this proposal would form a separate installation of software.

Message Handling

The python logging system implements the following structure:

in -> Manager -> Logger -> Handlers -> Formatter -> out

The "out" side feeds into the hub. The hub is also capable of feeding messages back into itself. The hub would be implemented as the following structure:

               External Message Generation
          +----------+         +----------+
          | Receiver | .. N .. | Receiver |
          +----------+         +----------+
+-------------+    +-----------+         +-------+
| Config Port |--->|           |<--------| Timer |
+-------------+    |    Hub    |         +-------+
       +-----------|           |<-------+
       |           +-----------+        |
       |                  |         +-----------------+
       |                  |         | LoopbackHandler |
       V                  V         +-----------------+
   +--------+         +--------+        |
   | Action | .. N .. | Action |--------+ new LogRecord
   +--------+         +--------+
       | new LogRecord |      ^
       |               |      |
       V               V      v
  +---------+   +---------+  +-------+
  | Handler |   | Handler |  | Store |
  +---------+   +---------+  +-------+
  | Client |

The LogRecord and Handler implementations are used from the PEP 282 implementation. Additional Handlers will be implemented as required. Filters are available at the Receivers, Hub, Actions and Handlers.


Messages come in from the various Receivers.

There will be an internal loopback Handler/Receiver which internal components may forward new LogRecords to as appropriate. These messages are fed straight back into the Hub.

A receiver may optionally use Filters before passing the LogRecord and to the hub.

The timer is a special receiver that just generates its own time LogRecords every second. These could be used by Actions which are checking state regularly.

Hub ...

The hub accepts the messages from the Receivers. It may perform three operations on the message:

  1. run the message through its internal filters

  2. use the message to alter its configuration

  3. pass the message on to Actions

The message may be blocked at step 1 or 2. The message may be altered at step 1 and passed on, just like any other LogRecordFilterer.


Actions are very much like a Logger that just passes through existing LogRecords, rather than generating them. Actions will be able to:

  • forward LogRecords to one or more Handlers

  • generate new, possibly replacement LogRecords (escallations, collations)

  • have state to draw upon

Several levels of complexity of Action exist:

  1. A standard Action class will be provided that is customisable using class attributes to define the tests being performed and the actions to take if the tests are positive.

  2. More complex Action classes will provide custom Handlers which in turn have Filters which mess with the record as it's being passed through.

  3. Really complex Action classes will override handleLogRecord() to utilise some state that is stored on disk.

Actions will be able to perform some standard message handling functions:


are possible when an Action recognises that a LogRecord has been sent into the hub which has not has some action perfomed on it (eg. a matching LogRecord which signifies that some action has taken place) within some time period.


are possible when an Action is set to recognise certain LogRecord patterns and discard duplicates of LogRecords within certain time periods.

See the Use Case Scenarios for more examples and detail.


Filters are available to the Receivers, Hub, Actions and Handlers. They may

  • reject a message outright

  • modify a message

Basic Filters are implemented in the logging module.


Clients, such as UIs, will be able to connect to the hub via the Config Port to indicate that a new forward should be set up to the client:

  1. hub is started with default configuration from config file

  2. client connects and in that connection message indicates the new configuration the hub should set up for it

  3. client may alter that configuration at any time

Closing the client-server connection may occur in the following situations:

  1. client disconnects (intentionally or otherwise) and hub removes all facilities set up for client

  2. server disconnects (intentionally or otherwise) and the client indicates this to the user. Client may additionally poll for server re-start.

Config Port (or "dedicated Hub Listener")

The hub and client configuration is performed through the config port. It's just another Receiver, really, but is always started.

The Hub will listen for CONFIG messages that indicate configuration changes.

Configuration messages may result in any of the following:

  • new Actions for clients

  • removed Actions for clients

  • new inbound Filters being added

  • inbound Filter rules being modified (e.g. squash filter)

The CONFIG messages are described in the Configuration section.


The configuration of the hub will ultimately determine whether the messages are:

The initial configuration will describe:


The receivers active in the hub and what they're listening to.


The actions that are set up by default.


The client configuration - who is allowed to connect and how. Some sort of access control spec.

At the moment, it's looking like the configuration will be done in a python module that actually launches the service/daemon.

Simple Configuration (minus imports):

hub = logext.Hub()

h = logging.StreamHandler()
f = logging.Formatter("%(asctime)s %(name)-19s - %(message)s")
f = logging.Filter()
f.level = ['ERROR']
a = logext.Action()

def tcpReceiver():
    r = logext.TCPReceiver(hub, 'localhost', logging.DEFAULT_TCP_LOGGING_PORT)
    print 'tcp started'

def udpReceiver():
    r = logext.UDPReceiver(hub, 'localhost', logging.DEFAULT_UDP_LOGGING_PORT)
    print 'udp started'

The receivers are now started in threads (see

The basic Receiver, Hub, Action, Filter and Handler objects will be configurable using a simple XML configuration file:

 <receiver type="TCP">
   <port> DEFAULT </port>
   <hostname> localhost </hostname>

 <receiver type="TCP">
   <port> DEFAULT </port>
   <hostname> remotehost </hostname>

   <level> CRITICAL </level>
  <handler type="File">
   <filename> all_critical.log </filename>

Additionally the client CONFIG messages may include remove commands:

  <action id="[id]"/>
  <handler id="[id]"/>

Replace the nominated Handler with this one:

 <handler type="File" id="[id]">
  <filename> alternative.log </filename>

Set up a new client connection with all INFO messages:

   <level> INFO </level>
  <handler type="Client"/>

Use Case Scenarios

GUI monitoring of log messages

Several systems generate logging messages which are collected in a central place. Operations personnel have a monitor client on their desktop which displays messages pertinent to them.

The events for a GUI connection look like:

client config message

Message comes in through the Config Port with information describing the message filtering that the client wishes to set up.

hub creates new Action

New Action is created with appropriate filters

client connected to Action

Hub passes connection from config port to action's handler. This part is a little (ok, a lot ;) hazy.

SMS alerting of critical messages

A series of daemon restarts in succession alerts on-call operators via SMS that something very bad is happening.

Remote site monitoring

A hub at the remote site filters the messages. It passes the filtered messages on to a hub at the local site for operations staff to monitor. The remote site hub may even take remedial action based on log messages.

Clients cancelling escallations and silencing messages for a period

Clients will want to be able to indicate to the Hub that a message has been dealt with, and that it should not escallate. Also, they will want to be able to squash messages for a period of time (expected downtime, or "retroactively scheduled downtime .... I'm trying to fix the %@#$% thing will someone shut the @#$%@!#$ escallation system up")

The events for escallation cancellation look like:

initial message

message comes in and an Action recognises it as something that should be actioned within 5 minutes or an escallation should be generated. It stores the time of the message off in a state file.


a client indicates to the Action that the message should be


five minutes pass (300 ticks of the Hub's timer) and nothing has happened, so the Action fires an escallation message off to its Handler (possibly email, possibly SMS, possibly voicemail playback)

The events for escallation squash are:

initial message

as above

squash message

CONFIG message comes in on the Hub's dedicated config port and is intercepted by the Hub. The message is decoded to extract the squash filter parameters and duration. A squash filter is then set up which destroys all messages of the given type until the duration passes. The duration is specified by the end filter parameter, and once it's passed the Hub is able to remove the Filter.



See for an initial implementation that provides the Hub, Action and Receiver base classes. It also implements TCP and UDP receivers.

It uses a modified and

Test it with the server. Send messages to the server with (which uses logging.cfg).

TODO: the default Filter described here could probably be made the default filter in the module...

TODO: does the Filter start/end refer to time.time() or record time?

TODO: we need to be able to ID Filters and Actions to configure them on the fly!


[PEP282] A Logging System

Python Logging System:



Thanks to Trent Mick, Vinay Sajip, Anthony Baxter and Dave Cole for their inspiration, code hackery and feedback on this proposal.

HTML generated from log_extensions.txt on 2002-03-28 by Docutils.