Ninja Scroll Logging Engine

This section describes the IO Ninja’s advanced logging engine from a programmer’s point of view.

In order to be able to work with truly huge logs, IO Ninja keeps all the log data in disk files, mapping and caching portions of log files as necessary. With this design, the size of a log is only limited by the amount of free disk space.

Log Files

A record file is the main log file. There are also derivative index file and fold file maintained for efficient navigation and filtering, but these are temporary and can always be re-created from the main record file.

The record file starts with a header log.RecordFileHdr. It describes meta-information about a log file and is directly followed by a list of log records (log.Record). Each record contains:

  1. Timestamp;
  2. Record code;
  3. Code-specific data (optional);

A record code is a 64-bit integer which specifies the kind of a record. 64-bits give enough space to generate record codes more-or-less automatically – and at the same time, avoid unintended collisions. A good approach to choosing record codes is to take timestamps measured in 100-nanosecond interval (aka, Windows FILETIME). Technically speaking, there is still a small chance for collisions; later we plan to run a free service for registering IO Ninja log record codes, thus eliminating the possibility of collisions completely.

Representers

To convert a binary record from a log file into human-readable representation, a user-defined representer is invoked. Representer must be a so-called pure function. What it means is that its output should only depend on the contents of a record — and nothing else. The signature of a representer function must conform to log.RepresenterFunc.

A representer function analyzes the contents of a log record (log.Record) and then creates a desired representation by setting attributes and calling methods of the log.Representation object passed as the first argument.

Example:

bool representFpgaUploaderLog(
    log.Representation* representation,
    uint64_t recordCode,
    void const* p,
    size_t size,
    uint_t foldFlags
) {
    switch (recordCode) {
    case FpgaUploaderLogRecordCode.StillWaiting:
        representation.m_lineAttr.m_backColor = ui.StdColor.PastelGray;
        representation.addHyperText("...still waiting...");
        break;

    case FpgaUploaderLogRecordCode.FpgaDisabled:
        representation.m_lineAttr.m_iconIdx = log.StdLogIcon.Info;
        representation.m_lineAttr.m_backColor = log.StdLogColor.Info;
        representation.addHyperText("FPGA disabled");
        break;

    ...

    default:
        return false;
    }

    return true;
}

Filters

A log filter does what you would expect it to do – it analyzes a log record (log.Record) and then returns true to show it, or false to hide it. Unlike representers, filters can be stateful. As such, they are defined as class objects (and not pure functions). The base class for all filters is log.Filter.

Example:

class EthernetTapLogFilter: log.Filter {
    io.PcapFilter m_pcapFilter;

    override bool filter(
        uint64_t timestamp,
        uint64_t recordCode,
        void const* p,
        size_t size
    ) {
        switch (recordCode) {
        case EthernetTapLogRecordCode.Packet_ch1:
        case EthernetTapLogRecordCode.Packet_ch2:
            return m_pcapFilter.match(p + MetaSize, size - MetaSize);

        default:
            return true; // everything else is visible
        }
    }
}

Converters

A log converter transforms each record from the original log file into zero or more records. It is used for doing complex transformations of the log, e.g., performing protocol analysis, complex filtering, highlighting, etc.

Converters are usually stateful; the base class for all converters is log.Converter. The convert method can return false to indicate that the record was not processed and should be kept as-is in the output log.

Converters can be pretty complex; for the demonstation purposes, below is a rather artificial example of a converter which doubles each log record.

Example:

class MyConverter: log.Converter {
    override bool convert(
        log.Writer* writer,
        uint64_t timestamp,
        uint64_t recordCode,
        void const* p,
        size_t size
    ) {
        writer.write(timestamp, recordCode, p, size);
        writer.write(timestamp, recordCode, p, size);
        return true;
    }
}

For a real-life examples of log converters, check the sources of Modbus Analyzer or Regex Colorizer plugins.

Range Processors

A range processor runs over the representation of a selection in the log – all while calculating some stats (checksums, throughputs, etc). Usually, this information is then displayed in the information grid (ui.InformationGrid) or the status bar (ui.StatusBar).

For real-life examples a rangle processors, check the sources of log.ChecksumCalcRangeProcessor or log.ThroughputCalcRangeProcessor.