Note some of the components described in this document are implemented in the python logging system. In the course of designing this system, some changes were proposed to the logging implementation. These changes have been accepted but not implemented, so some of the implementation details described here will be out of sync with the reality of the logging system for a while.
The hub and UI would perform the following duties:
accept messages from the logging module via the logging module's handlers (FileHandler, SocketHandler, SyslogHandler, SMTPHandler, NTEventLogHandler, SQLHandler)
redistribution of messages to other targets (just plucking ideas from the ether here) ... all the above Handlers, ElvinHandler, OpenviewHandler, SunViewManagerHandler, JabberHandler, SMSHandler
perform some action on incoming messages, possibly aggregating, dropping or forwarding
have state which would be affected by the actions and may cause new messages to be generated (ie. escalation)
present a display of the messages to a user, either graphically or textually (the most basic of these would resemble the UNIX tail -f)
Note: it is not the intention of this proposal to implement a monitoring system like EDDIE. Monitoring of systems is performed by other components which generate log messages. It is the intention of this proposal to present those messages to users in meaningful and useful ways.
Note: it is not the intention of this proposal to add this functionality to the core logging implementation of PEP 282. The implementation of this proposal would form a separate installation of software.
The python logging system implements the following structure:
in -> Manager -> Logger -> Handlers -> Formatter -> out
The "out" side feeds into the hub. The hub is also capable of feeding messages back into itself. The hub would be implemented as the following structure:
External Message Generation | V +----------+ +----------+ | Receiver | .. N .. | Receiver | +----------+ +----------+ | V +-------------+ +-----------+ +-------+ | Config Port |--->| |<--------| Timer | +-------------+ | Hub | +-------+ +-----------| |<-------+ | +-----------+ | | | +-----------------+ | | | LoopbackHandler | V V +-----------------+ +--------+ +--------+ | | Action | .. N .. | Action |--------+ new LogRecord +--------+ +--------+ | new LogRecord | ^ | | | V V v +---------+ +---------+ +-------+ | Handler | | Handler | | Store | +---------+ +---------+ +-------+ | V +--------+ | Client | +--------+
The LogRecord and Handler implementations are used from the PEP 282 implementation. Additional Handlers will be implemented as required. Filters are available at the Receivers, Hub, Actions and Handlers.
Messages come in from the various Receivers.
There will be an internal loopback Handler/Receiver which internal components may forward new LogRecords to as appropriate. These messages are fed straight back into the Hub.
A receiver may optionally use Filters before passing the LogRecord and to the hub.
The timer is a special receiver that just generates its own time LogRecords every second. These could be used by Actions which are checking state regularly.
The hub accepts the messages from the Receivers. It may perform three operations on the message:
run the message through its internal filters
use the message to alter its configuration
pass the message on to Actions
The message may be blocked at step 1 or 2. The message may be altered at step 1 and passed on, just like any other LogRecordFilterer.
Actions are very much like a Logger that just passes through existing LogRecords, rather than generating them. Actions will be able to:
forward LogRecords to one or more Handlers
generate new, possibly replacement LogRecords (escallations, collations)
have state to draw upon
Several levels of complexity of Action exist:
A standard Action class will be provided that is customisable using class attributes to define the tests being performed and the actions to take if the tests are positive.
More complex Action classes will provide custom Handlers which in turn have Filters which mess with the record as it's being passed through.
Really complex Action classes will override
handleLogRecord() to utilise
some state that is stored on disk.
Actions will be able to perform some standard message handling functions:
are possible when an Action recognises that a LogRecord has been sent into the hub which has not has some action perfomed on it (eg. a matching LogRecord which signifies that some action has taken place) within some time period.
are possible when an Action is set to recognise certain LogRecord patterns and discard duplicates of LogRecords within certain time periods.
See the Use Case Scenarios for more examples and detail.
Filters are available to the Receivers, Hub, Actions and Handlers. They may
reject a message outright
modify a message
Basic Filters are implemented in the logging module.
Clients, such as UIs, will be able to connect to the hub via the Config Port to indicate that a new forward should be set up to the client:
hub is started with default configuration from config file
client connects and in that connection message indicates the new configuration the hub should set up for it
client may alter that configuration at any time
Closing the client-server connection may occur in the following situations:
client disconnects (intentionally or otherwise) and hub removes all facilities set up for client
server disconnects (intentionally or otherwise) and the client indicates this to the user. Client may additionally poll for server re-start.
The hub and client configuration is performed through the config port. It's just another Receiver, really, but is always started.
The Hub will listen for CONFIG messages that indicate configuration changes.
Configuration messages may result in any of the following:
new Actions for clients
removed Actions for clients
new inbound Filters being added
inbound Filter rules being modified (e.g. squash filter)
The CONFIG messages are described in the Configuration section.
The configuration of the hub will ultimately determine whether the messages are:
forwarded on to other logging mechanisms
forwarded on to clients
filtered for manipulation or discarding
The initial configuration will describe:
The receivers active in the hub and what they're listening to.
The actions that are set up by default.
The client configuration - who is allowed to connect and how. Some sort of access control spec.
At the moment, it's looking like the configuration will be done in a python module that actually launches the service/daemon.
Simple Configuration (minus imports):
hub = logext.Hub() h = logging.StreamHandler() f = logging.Formatter("%(asctime)s %(name)-19s - %(message)s") h.setFormatter(f) f = logging.Filter() f.level = ['ERROR'] a = logext.Action() a.addHandler(h) a.addFilter(f) hub.addAction(a) def tcpReceiver(): r = logext.TCPReceiver(hub, 'localhost', logging.DEFAULT_TCP_LOGGING_PORT) print 'tcp started' r.serve_forever() def udpReceiver(): r = logext.UDPReceiver(hub, 'localhost', logging.DEFAULT_UDP_LOGGING_PORT) print 'udp started' r.serve_forever()
The receivers are now started in threads (see test_hub.py).
The basic Receiver, Hub, Action, Filter and Handler objects will be configurable using a simple XML configuration file:
<hub> <receiver type="TCP"> <port> DEFAULT </port> <hostname> localhost </hostname> </receiver> <receiver type="TCP"> <port> DEFAULT </port> <hostname> remotehost </hostname> </receiver> <action> <filter> <level> CRITICAL </level> </filter> <handler type="File"> <filename> all_critical.log </filename> </handler> </action> </hub>
Additionally the client CONFIG messages may include remove commands:
<hub> <remove> <action id="[id]"/> <handler id="[id]"/> </remove> </hub>
Replace the nominated Handler with this one:
<hub> <handler type="File" id="[id]"> <filename> alternative.log </filename> </handler> </hub>
Set up a new client connection with all INFO messages:
<hub> <action> <filter> <level> INFO </level> </filter> <handler type="Client"/> </action> </hub>
Several systems generate logging messages which are collected in a central place. Operations personnel have a monitor client on their desktop which displays messages pertinent to them.
The events for a GUI connection look like:
Message comes in through the Config Port with information describing the message filtering that the client wishes to set up.
New Action is created with appropriate filters
Hub passes connection from config port to action's handler. This part is a little (ok, a lot ;) hazy.
A series of daemon restarts in succession alerts on-call operators via SMS that something very bad is happening.
A hub at the remote site filters the messages. It passes the filtered messages on to a hub at the local site for operations staff to monitor. The remote site hub may even take remedial action based on log messages.
Clients will want to be able to indicate to the Hub that a message has been dealt with, and that it should not escallate. Also, they will want to be able to squash messages for a period of time (expected downtime, or "retroactively scheduled downtime .... I'm trying to fix the %@#$% thing will someone shut the @#$%@!#$ escallation system up")
The events for escallation cancellation look like:
message comes in and an Action recognises it as something that should be actioned within 5 minutes or an escallation should be generated. It stores the time of the message off in a state file.
a client indicates to the Action that the message should be
five minutes pass (300 ticks of the Hub's timer) and nothing has happened, so the Action fires an escallation message off to its Handler (possibly email, possibly SMS, possibly voicemail playback)
The events for escallation squash are:
CONFIG message comes in on the Hub's dedicated config port and is intercepted by the Hub. The message is decoded to extract the squash filter parameters and duration. A squash filter is then set up which destroys all messages of the given type until the duration passes. The duration is specified by the end filter parameter, and once it's passed the Hub is able to remove the Filter.
See logext.py for an initial implementation that provides the Hub, Action and Receiver base classes. It also implements TCP and UDP receivers.
TODO: the default Filter described here could probably be made the default filter in the logging.py module...
TODO: does the Filter start/end refer to time.time() or record time?
TODO: we need to be able to ID Filters and Actions to configure them on the fly!
http://python.sourceforge.net/peps/pep-0282.html: A Logging System
Python Logging System: http://www.red-dove.com/python_logging.html
Thanks to Trent Mick, Vinay Sajip, Anthony Baxter and Dave Cole for their inspiration, code hackery and feedback on this proposal.
This document has been placed in the public domain.
HTML generated from
log_extensions.txt on 2002-03-28 by Docutils.