Advanced logging systems

is always nice to have a good logging system, that will help programmer to understand what happened in case of crash or non proper behaviour. Well, if it crashed it is usually quite easy to find a place where it crashed, but lets assume some part of your system misbehaved that cause misbehaviour of another system. This situation still OK if you can debug your program and know exactly the preconditions. But lets assume you are running the production system, so you have no a debug log switched on, cause otherwise your system will slow down because of logging, and even in case debug level of logging is switched on, in case when one of 1000 requests will fail, it will be a hell to find proper debug logs among others.

The solution could be in writing the advanced log. What I mean as advanced log is the following:

1) For the normal processed flow we have a logger that accumulates in memory all debug information, that could be cheap in such languages as erlang where it is cheap to create additional process.

2) For normal processed flow you write only one line to the log file, that will still show you the final state of your flow and that all was processed as expected.

3) in case something failed ( put log to the logger with warning level or even error level), logger can put to the file all the accumulated information that system put to it regarding this business process on debug level, and starting from this moment it switched to write all debug messages from this business process.

If you managed to follow this simple 3 rules ( that is will be not so obvious in some of the languages), you can be assured that you’ll be able to understand what part of the system misbehaved. In case you have enough debug information (and as some project shows, log code could take up to 1/4 of the entire code), it is very easy to maintain the system.

Together with the hot code swapping as for instance erlang language has, you can implement a hot fix quite fast and apply it to the running instance of your system(s).