IO Ninja 5.5.0 Is Out!
Major release

• New fast Regex Markup engine (based on a heavily modified fork of the Google RE2 library)
• 100% accurate display of all UTF-8 continuation bytes, unprintable characters, and invalid or incomplete UTF-8 multi-byte sequences across line and page boundaries in the log and hex editor
• New implementation of the Plain Text binary data view in log with accurate line wraps in the middle of invalid UTF-8 multi-byte sequences (no more overhangs!) and optional visibility of CR, LF, and Tab characters
• New built-in Jancy type string_t (non-zero-terminated UTF-8 string slice) is now used instead of char const* for strings in all scripts
• New implementation of the Jancy regex switch statement and regex match operator =~ (based on the very same fork of Google RE2)
• Release of the Device Monitor for Linux (tdevmon-3.3.13) with support for the latest Linux kernels
• Numerous critical fixes in various modules
  • Reverse Text Find crashes and missed matches
  • Crashes in the Terminal on VT102 status requests with the Local Echo mode on
  • Crashes when dragging packets from Packet Library panes to the Transmit pane
  • Deadlocks when adding/removing layers with an active onLogRecord() handler
  • Pasting text to the Text Transmit pane doubled all backslashes
  • Visual glitches when adding or deleting multiple lines of data in the hex editor
  • Critical updates in the ARM32 & ARM64 ABI compatibility in the Jancy scripting engine

New Regex Markup Engine

The original Regex Markup engine introduced a year ago was a breakthrough — the ability to highlight important chunks of data and automatically split the incoming stream of data into packets was truly appreciated by our users. We kept working on the engine and eventually, after numerous benchmarks and tests, decided to rebase it on the Google RE2 — one of the fastest industry-standard cross-platform libraries for regular expression processing.

The problem, however, was that Google RE2 doesn't support the streaming mode — which is essential for using it in a live IO environment, where data arrives incrementally, chunk-by-chunk. To combat this limitation, we had to go for some in-depth modifications and introduce an entirely new stream-matching API for RE2. That wasn't easy, but the result was worth it. The new Regex Markup engine in IO Ninja got up to 3x faster, more robust, and it now supports everything that RE2 does (including whole-word and case-insensitive options missing in the original implementation).

Try it, and you will love to use it!

Improved Binary Data View

Binary data handling is where IO Ninja always shined. And with this release, we put significant efforts into making it even better.

First, all special control characters are now mapped to their standard visual representations (☺ for SOH, ☻ for STX, • for BEL, ‹ for UTF-8 continuation character, etc). This provides important visual clues when analyzing hex dumps. For those users who prefer the original (somewhat less cluttered) view where all unprintables are simply replaced with a . (dot) — there's an option for that, too.

Secondly, the "Plain Text" view in the log is fully reimplemented and now offers accurate text wrapping — even in the middle of invalid UTF-8 sequences (which previously resulted in overhangs). Also, there's now an option to display special whitespace characters CR, LF, and Tab (previously, CR was shown as ., LF was hidden, and Tab was expanded to spaces). All the special control character mappings mentioned above apply to the "Plain Text" view, as well.

Jancy Enhancements: string_t, Regex switch, Regex Match Operator =~

While not that apparent at first glance, this is a big one! All IO Ninja scripts now use the new Jancy type string_t (a non-zero-terminated UTF-8 string slice) instead of char const* (a pointer to a zero-terminated UTF-8 string) for string data. This improves efficiency by avoiding unnecessary memory allocations, enables string-specific operators, and makes the APIs cleaner and more explicit.

Also, there's a new and improved implementation of the regex switch and the regex match operator =~ (based on the very same fork of Google RE2 as used for the Regex Markup engine in IO Ninja). Both now support case-insensitive and whole-word options and perform matching up to 3x faster than before.

string_t packet = readPacket();

// use the regex match operator to check if a packet matches...
// ...and extract the goodies if it does
if (packet =~ "\x02(.*?)\r")
    print($"in-band command detected: $1\n");

string_t packet = readPacket();

// efficiently parse the packet using a regex DFA
[ RegexCaseInsensitive ] // allow case-insensitive matching
switch (packet) {
case r"open\s+([a-z0-9]*)":
    print($"open device: $1\n");
    break;
case r"set\s+([a-z0-9]+)\s*=\s*([a-z0-9]+)":
    print($"set value: $1 = $2\n");
    break;
// ...
}

Device Monitor for Linux 6.5 (Or Higher)

Keeping Device Monitor up-to-date with new Linux Kernel releases is a never-ending whack-a-mole game. In the releases linux-6.4 and then linux-6.5, the Kernel team introduced two breaking changes (namely, the change of signature for the class_create() function and the removal of the ITER_PIPE vectored IO) that caused tdevmon to stop compiling.

This release brings Device Monitor up to speed, and it's compatible with the latest Linux kernels once again (tested with the latest-and-greatest linux-6.7.0).

Numerous Critical Fixes

• Reverse Text Find crashes and missed matches

  The previous implementation of Text Find was prone to miss matches spanning across cache page borders — and even crash on some rare invalid UTF-8 multi-byte sequences. Fixed now.

• Crashes in the Terminal on VT102 status requests with the Local Echo mode on

  Unintentional VT102 status-request sequences in a terminal with the Local Echo mode on could have led to a crash. Local Echo is rarely used; that's why this issue stayed under the radar for so long. Fixed now.

• Crashes when dragging packets from Packet Library panes to the Transmit pane

  A regression introduced two versions ago caused a crash when dragging a library packet into the Transmit pane for editing. Fixed now.

• Deadlocks when adding/removing layers with an active onLogRecord() handler

  This rare deadlock condition could have arisen when attempting to manage the session layers while running a script that invokes transmit from within the onLogRecord() entry point. Fixed now.

• Pasting text to the Text Transmit pane doubled all backslashes

  Previous versions of IO Ninja tried to escape-encode the data pasted into the Text Transmit pane. This caused unexpected (and unwanted!) effects when the original string was itself an escape-encoded string. The pre-paste escape-encoding is disabled now.

• Visual glitches when adding or deleting multiple lines of data in the hex editor

  Under certain conditions, multi-line editing operations in the hex editor failed to update the vertical scroll bar properly (and could leave some visual artifacts). Fixed now.

• Critical updates in the ARM32 & ARM64 ABI compatibility in the Jancy scripting engine

  These ABI incompatibilities only started to manifest themselves after switching to string_t in scripts, after the signatures of many functions had changed (so it should not have caused any crashes in the previous versions of IO Ninja). Nevertheless, any ABI incompatibilities are critical and must be ironed out. Which they are now.