IO Ninja 5.6.0 Is Out!
Major release

• New syntax constructs dylayout and dyfield for describing dynamic layouts (i.e., data structures defined at runtime)
• Major improvements in the Packet Template engine — dynamic packet structures, byte-range colorization, support for bitflag enum fields, etc.
• New Modbus Analyzer plugin based on the new dynamic layout engine
• New Protocol Analyzer Plugin Wizard for creating your own protocol analyzers — all you have to do is redefine the packet layout function!
• New in-app scripting pretransmit entry point for the on-the-fly encoding and other modifications of outbound packets
• Standalone *.njlog logs can create toolbars now (e.g., logs created by Pcap Sniffer and Ethernet Tap now have filtering and export-to-pcap controls)
• Numerous critical fixes in various modules

Dynamic Layouts

One of the main reasons for creating the Jancy language was the desire to have a scripting language with safe and convenient facilities for handling raw binary data. Hence, Jancy provides C-compatible structs, unions, bit-fields, pointer arithmetics — and even native support for bigendian integers. All this allows first describing protocol data structures in a declarative way and then "walking" across binary buffers with pointer arithmetics ("overlapping" portions of the buffer with the protocol data struct pointers). As a result, you can access all protocol data fields naturally and efficiently — without having to painstakingly glue them back together from raw bytes (unlike in other languages).

For most real-world protocols, the structure of a packet is not static — it depends on the preceding fields. And while C-structs and pointer arithmetics make parsing binary blobs much easier, there are still a few non-trivial tasks that need to be carefully taken care of:

1. Buffering the necessary amount of bytes before attempting to access data fields via struct pointers;
2. Advancing the data pointer properly when moving on to the next portion of the packet;
3. If we buffered the first portion of a packet, but the next portion of the packet requires more bytes than we already have — we have to stop parsing, buffer more bytes, and then pick up where we left off;
4. If we want to render the structure of the parsed packet later on (e.g., in the log or the property grid), we also have to maintain a list of all discovered fields as we parse.

Aiming to have the perfect protocol analyzer platform, we were determined to improve the situation. And after a lot (a lot!) of experiments and tests, we finally managed to find an approach that solves all those issues! I know it could sound almost too good to be true, but all you have to do is write the packet layout function:

async layoutMyProto(jnc.DynamicLayout* layout) {
    dylayout (layout) {
        dyfield MyProtoHdr hdr; // map ProtoHdr as a dynamic field
        switch (hdr.m_command) { // depending on m_command, define follow-up fields
        case MyProtoCmd.Cmd1:
            dyfield MyProtoCmd1 cmd; // Cmd1-specific data
            break;

        case MyProtoCmd.Cmd2:
            dyfield MyProtoCmd2 cmd; // Cmd2-specific data
            dyfield char payload[cmd.m_dataLength]; // dynamically sized array
            break;

        // ...and so on
        }
    }
}

After that, IO Ninja will be able to parse the packet as specified (automatically pausing and waiting for more bytes when needed). If this function is called from the protocol analyzer plugin, all discovered data fields will be rendered in the log view as a tree. The same function can also be used as a packet template, too (thus letting you conveniently fill in all the packet fields via a property grid).

The scale of this update may not be obvious at first glance. But in reality, this is a huge step forward that will allow the effortless creation of custom protocol analyzers in IO Ninja.

For a real-world example, check the actual implementation of the Modbus PDU layout function used in the official Modbus Analyzer plugin.

Packet Template Enhancements

The Packet Template engine is a unique feature of IO Ninja that made life much easier for those who had to work with non-trivial protocols (such as Modbus). The ability to conveniently fill bigendian integers and bit-fields using a property grid and automatically calculate CRC checksums before transmission was truly appreciated by our users.

With this release, packet templates received a major boost — courtesy of the new dynamic layout facility. Now, packets can dynamically change their structure — depending on what you select in the property grid!

Protocol analyzer plugins can now add packet templates of their own. Representation of a packet in the log and in the property grid is handled by the very same layout function, so plugins can kill two birds with one stone!

Individual fields can now automatically convert to & from strings — for example, an IPv4 address will be represented and edited as a single string like 192.168.1.34 instead of four integer values.

Another noteworthy addition is the ability of the layout function to colorize portions of the packet — besides being an eye-candy, it also provides important visual clues when switching between the property grid and the hex editor.

Last but not least, packet templates now represent bitflag enum fields as a series of checkbox values to allow you conveniently turn individual bits on and off.

New Modbus Plugin

The Modbus Analyzer plugin received a major makeover, as well. It utilizes the new dynamic layout engine and its ability to assign colors to different packet portions.

The new Modbus Analyzer plugin also includes packet templates for Modbus RTU and Modbus TCP — unlike before, now there's no need to separately load and select the corresponding packet template script.

There's also a new stock modbus-ascii.jnc script that takes a Modbus RTU packet (prepared, for example, using the built-in packet template and the property grid) and re-encodes it as Modbus ASCII. This script utilizes the new pretransmit facility described below.

New pretransmit Entry Point

Now, it's possible to modify the outbound packets via the pretransmit function in the Script pane. When your script is running, every time you click Send in the Transmit pane, the packet will be dispatched through your pretransmit function and processed accordingly.

This is a powerful and easy-to-use feature; here are a few examples to get you started.

void pretransmit(
    char const* p,
    size_t size
) {
    char* p2 = new char[size]; // allocate a buffer for the modified packet

    // reverse the original packet before transmission...
    for (size_t i = 0, j = size - 1; i < size; i++, j--)
        p2[i] = p[j];

    transmit(p2, size); // ...and out it goes!
}

void pretransmit(
    void const* p,
    size_t size
) {
    transmit("\x02"); // STX
    transmit(p, size);
    transmit("\r\n"); // CR LF
}

void pretransmit(
    char const* p,
    size_t size
) {
    for (size_t i = 0; i < 3; i++) { // re-transmit 3 times
        transmit(p, size);
        sys.sleep(500); // wait for half a second and repeat
    }
}

import "crc16.jnc"

void pretransmit(
    char const* p,
    size_t size
) {
    transmit(p, size); // first, the original data...
    bigendian uint16_t crc = crc16_ansi(p, size);
    transmit(&crc, sizeof(crc)); // ...then, its CRC
}

Numerous Critical Fixes

• Some UI form controls became irresponsive after a while and could crash when clicked

  UI form objects were not added as GC roots and thus could be incorrectly garbage-collected. After that it became not reponsive — and in rare instances could even cause a crash of the ioninja-server process. Fixed now.

• A buffer overrun could cause crashes during Serial port enumeration on macOS

  Depending on the model of USB-to-Serial adapters plugged into your Mac, enumeration of serial devices in the Serial plugin could have caused a crash; typically, the crash occurred right during plugin startup. Fixed now.

• A subtle off-by-one buffer overrun could cause crashes on 32-bit platforms

  This overrun could only manifest itself on 32-bit platforms (both Intel and ARM) and required somewhat rare preconditions from the C runtime, so most likely, not many users (if any) fell victim to this bug. Nevertheless, a buffer overrun is always a critical bug, and it's fixed now.

• Invalid signature for the implementation of std.setError(string_t) could cause crashes on adding errors to the log

  This regression was introduced when switching from char const* to string_t a few releases ago (the API changed, but the underlying implementation didn't). This bug could result in an incorrect display of error messages in the log and could cause a crash on rare occasions. Fixed now.

• ...and many others