Dear friends!

It has been a long time since we released an update for IO Ninja -- more than eight months! Such a delay happened because we were working hard on the new major release, IO Ninja 5. And I'm happy to announce that it's finally out!

The most significant change from the previous generation of IO Ninja is the introduction of accounts, capabilities, and subscriptions. Please read the intro on our website to learn more about our motivation for such a major change. The dedicated section of the user manual covers this new ecosystem in more detail.

The IO Ninja 5 also features two new plugins dedicated to helping web developers debug modern WebSocket-based applications. Please read more about these plugins on the respective pages: WebSocket Client and WebSocket Server.

IO Ninja 5 includes a critical update of the Device Monitor Service for Linux that enables running the Serial Monitor on modern Linux kernels. There's also a major clean-up of UI for the macOS version, critical bug fixes and optimizations in the logging engine, and plenty of other important updates and improvements.

Besides the software updates, with IO Ninja 5, we introduce a new e-store that seamlessly integrates with the main website and provides a much more smooth experience. And of course, we -- finally! -- opened this forum, just as so many of you suggested. Here, you can ask questions, share your suggestions on improving IO Ninja, and receive support -- both from the IO Ninja Team and other users.

We worked with passion and dedication to release this new version of IO Ninja. We are sure that IO Ninja will help you in your work, and you will enjoy using it!

Currently, IAS (in-app-script) can only control the underlying session via:

• connect()
• disconnect()
• transmit(p, size)

Fine-tuning of the sessions configuration is currently not possible from IAS. But we plan to add this capability in the near future.

Each session class will export a public IAS interface (e.g., a serial session will have properties to query and control baud rate, parity, status lines, etc.) Then, IAS should be able to access this interface via a global constant g_session (or something like that).

Hello Bartosz,

The script for a filter to do what you want is very simple. First, you deduce the state of the CS line (from the I2cSpiTapLogRecordCode.SpiStart and SpiStop log records); then, you use this state to either hide or show the MOSI/MISO data (the log.StdRecordCode.TxRx log records).

The source code for such a filter might look something like this:

import "doc_Layer.jnc"
import "I2cSpiTap/I2cSpiTapLogRecordCode.jnc"

class SpiCsFilterLayer:
	doc.Layer,
	log.Filter
{
protected:
	ui.EnumProperty* m_csFilterProp; // a property to choose the filtering strategy
	int m_cs; // the state of the CS line

public:
	construct(doc.PluginHost* pluginHost);

	override bool filter(
		uint64_t timestamp,
		uint64_t recordCode,
		void const* p,
		size_t size
		);
}

SpiCsFilterLayer.construct(doc.PluginHost* pluginHost)
{
	basetype.construct(pluginHost);

	ui.EnumPropertyOption csFilterOptions[] = {
		{ "Show always", -1 },
		{ "Show when CS low", 0 },
		{ "Show when CS high", 1 },
	}

	m_csFilterProp = m_pluginHost.m_propertyGrid.createEnumProperty(
		"MOSI/MISO filter",
		"Show MOSI/MISO filtering criteria",
		csFilterOptions,
		countof(csFilterOptions)
	);

	m_cs = -1; // -1 means unknown
	pluginHost.m_log.addFilter(this);
}

bool SpiCsFilterLayer.filter(
	uint64_t timestamp,
	uint64_t recordCode,
	void const* p,
	size_t size
	)
{
	bool isVisible = true;

	switch (recordCode)
	{
	case log.StdRecordCode.SessionStarted:
		m_cs = -1; // reset to unknown
		break;

	case I2cSpiTapLogRecordCode.SpiStart:
		m_cs = 0;
		break;

	case I2cSpiTapLogRecordCode.SpiStop:
		m_cs = 1;
		break;

	case log.StdRecordCode.TxRx:
		isVisible =
			m_csFilterProp.m_value == -1 || // show always
			m_csFilterProp.m_value == m_cs; // matches the current state of CS
		break;
	}

	return isVisible;
}

An archive with the complete filter plugin is attached (SpiCsFilter.7z)

Usage:

1. Start I2C/SPI session
2. Open the SpiCsFilter.njplg file to attach this filter
3. Go to "Settings"
4. Select the appropriate filtering criteria
5. Hit "Apply and rebuild log" to apply the filter to the whole log (clicking "Apply" or "OK" will apply the filter "from now on" (i.e., to the follow-up log records only)

Hello,

Apologies for the delayed response. Support for njlog output directly from ioninja-hwc is on our TODO list (it was already requested by users). For the time being, you need to post-process the raw output of ioninja-hwc using any scripting language of your choice.

The protocol of communication between ioninja-hwc and plugins such as Serial over SSH or Serial Tap over SSH is not currently documented, but the decoding process and all relevant constants and data structures can be looked up in the sources of those plugins (remember, all IO Ninja plugins are open-sourced and available at ioninja/scripts/plugins).

In particular, all the relevant constants and packet structures for the ioninja-hwc protocol are contained in ioninja/scripts/common/io_HwcProto.jnc. The output file generated by ioninja-hwc is basically a sequence of "out" messages of this protocol. Each message starts with HwcMsgHdr followed by extra data block, the meaning of which depends on the message code. For example, HwcMsgCode.Rx is followed by the received bytes, HwcMsgCode.SerialTapCtsDsr is followed by HwcSerialStatusLines flags, etc.

Try decoding the output of ioninja-hwc using any language of your choice, and feel free to let me know if you run into a stumbling block or have any other questions.

The issue with adding TCP/UDP Analyzer layers via the Layer Pipeline dialog is confirmed. Under these conditions, due to a bug, the filter gets inserted in between the Ethernet Tap and TCP/UDP Analyzer (instead, it must be attached after TCP/UDP Analyzer).

The issue will be fixed in the very next release. Meanwhile, you can add these layers by clicking the down arrow next to the Layer Pipeline toolbar button (a button with a plus icon):

When added like this, the address filter behaves as expected.

Am I right in thinking that we should be able to enter an IP address OR a TCP port number into that box?

Yes. In that box, you're supposed to enter an IP address with or without a TCP port -- or just a TCP port alone.

Whatever we enter we see no traffic at all.

Hmm, just did a quick test and it seems to work as expected. Could you please share your session (Save -> Save Session, then archive the session folder and attach it to your post) and the filter strings you tried to apply?

BTW, very minor nit, but the placeholder text in the filter box reads "Enter a filter andress...", not "Enter a filter address...".

A typo! Will be fixed.

Thank you for your feedback!

BTW the TX/RX Filter layer shouldn't have any effect on the Ethernet Tap or Pcap Sniffer sessions -- as there are no TX/RX data streams there; just the raw Ethernet packets.

But after reconstructing TCP or UDP conversations with TCP/UDP Analyzer, the TX/RX Filter layer can be applied.

Indeed, there's a bug in TCP/UDP analyzer scripts. The fix will be included in the next official release; meanwhile, please use the patch below:

scripts.7z

Simply unpack it overwriting all corresponding *.jnc files. Make sure you overwrite files, not simply unpack it in the scripts folder (thus creating scripts/scripts).

Hello,

Apologies for the delayed response. Support for njlog output directly from ioninja-hwc is on our TODO list (it was already requested by users). For the time being, you need to post-process the raw output of ioninja-hwc using any scripting language of your choice.

The protocol of communication between ioninja-hwc and plugins such as Serial over SSH or Serial Tap over SSH is not currently documented, but the decoding process and all relevant constants and data structures can be looked up in the sources of those plugins (remember, all IO Ninja plugins are open-sourced and available at ioninja/scripts/plugins).

In particular, all the relevant constants and packet structures for the ioninja-hwc protocol are contained in ioninja/scripts/common/io_HwcProto.jnc. The output file generated by ioninja-hwc is basically a sequence of "out" messages of this protocol. Each message starts with HwcMsgHdr followed by extra data block, the meaning of which depends on the message code. For example, HwcMsgCode.Rx is followed by the received bytes, HwcMsgCode.SerialTapCtsDsr is followed by HwcSerialStatusLines flags, etc.

Try decoding the output of ioninja-hwc using any language of your choice, and feel free to let me know if you run into a stumbling block or have any other questions.

Hmmm.. I installed exactly the same kernel as on your box:

vladimir@raspberrypi:~ $ uname -a
Linux raspberrypi 5.15.32-v7+ #1538 SMP Thu Mar 31 19:38:48 BST 2022 armv7l GNU/Linux

Then overwritten the contents of src/lkm with the archive I uploaded earlier. It works!

vladimir@raspberrypi:~/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm $ make
make -C /lib/modules/5.15.32-v7+/build/ M=/home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm modules
make[1]: Entering directory '/usr/src/linux-headers-5.15.32-v7+'
  CC [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/module.o
  CC [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/Device.o
  CC [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/Hook.o
/home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/Hook.c: In function ‘Hook_stop’:
/home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/Hook.c:143:22: warning: suggest parentheses around ‘&&’ within ‘||’ [-Wparentheses]
  143 |   self->m_fops->read && self->m_fops->read != Hook_fop_read ||
      |   ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/Hook.c:144:23: warning: suggest parentheses around ‘&&’ within ‘||’ [-Wparentheses]
  144 |   self->m_fops->write && self->m_fops->write != Hook_fop_write ||
      |   ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/Hook.c:145:27: warning: suggest parentheses around ‘&&’ within ‘||’ [-Wparentheses]
  145 |   self->m_fops->read_iter && self->m_fops->read_iter != Hook_fop_read_iter ||
      |   ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/Hook.c:146:28: warning: suggest parentheses around ‘&&’ within ‘||’ [-Wparentheses]
  146 |   self->m_fops->write_iter && self->m_fops->write_iter != Hook_fop_write_iter ||
      |   ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/Hook.c:147:32: warning: suggest parentheses around ‘&&’ within ‘||’ [-Wparentheses]
  147 |   self->m_fops->unlocked_ioctl && self->m_fops->unlocked_ioctl != Hook_fop_unlocked_ioctl ||
      |   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/Hook.c:148:30: warning: suggest parentheses around ‘&&’ within ‘||’ [-Wparentheses]
  148 |   self->m_fops->compat_ioctl && self->m_fops->compat_ioctl != Hook_fop_compat_ioctl
      |   ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  CC [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/Connection.o
  CC [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/HashTable.o
  CC [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/ScatterGather.o
  CC [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/FileNameFilter.o
  CC [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/lkmUtils.o
  CC [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/stringUtils.o
  LD [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/tdevmon.o
  MODPOST /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/Module.symvers
  CC [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/tdevmon.mod.o
  LD [M]  /home/vladimir/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm/tdevmon.ko
make[1]: Leaving directory '/usr/src/linux-headers-5.15.32-v7+'

vladimir@raspberrypi:~/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm $ sudo insmod tdevmon.ko
vladimir@raspberrypi:~/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/share/tdevmon/src/lkm $ 
vladimir@raspberrypi:~/Projects/ioninja/devmon/build/make-arm32/tdevmon-3.3.9-a-linux-arm32/bin $ sudo ./tdevmon -m /dev/ttyUSB0
Monitoring /dev/ttyUSB0 (module: usbserial), hit Ctrl+C to interrupt...

16:13:19.821 open
    pid:             2604
    tid:             2604
    result:          0
    file id:         00000000b902eb40
    flags:           00020241
    mode:            00000002
    file name:       /dev/ttyUSB0

16:13:19.821 write_iter
    pid:             2604
    tid:             2604
    result:          4
    file id:         00000000b902eb40
    offset:          0
    buffer size:     4
    data size:       4

    0000  61 62 63 0a                                      abc.

16:13:19.822 close
    pid:             2604
    tid:             2604
    result:          0
    file id:         00000000b902eb40

Are you sure you rebooted, or at the very least, removed the previous instance of tdevmon.ko via rmmod before testing? Also, please make sure you overwrite all source files, not simply unpack the archive into src/lkm (thus creating src/lkm/lkm).

Actually, to make things more straightforward, here's a re-packaged patch:

tdevmon-3.3.9-a-linux-arm32.tar.gz

Please let me know if it works (again, reboot before testing).

OK, thanks. I will try to match your Raspbian as close as possible and see if I get it.

@jeremy-herbert

I see what you mean.

First of all, we are not going to completely match the output of what pyserial gives you. They enumerate and extract all the information by traversing the Linux filesystem and parsing file names and file contents; in IO Ninja we talk to systemd directly via libudev which is more efficient and IMHO a better approach overall.

But we can definitely add extra pieces of information to the description. Here's what we get for free from libudev:

getSysPath: /sys/devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.1/1-3.1:1.0/ttyUSB0/tty/ttyUSB0
getSysName: ttyUSB0
getSysNum: 0
getDevPath: /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.1/1-3.1:1.0/ttyUSB0/tty/ttyUSB0
getDevNode: /dev/ttyUSB0
getDevType: 
getSubsystem: tty
getDriver: ch341-uart
getAction: 
property[CURRENT_TAGS] = ':systemd:'
property[DEVLINKS] = '/dev/serial/by-id/usb-1a86_USB2.0-Ser_-if00-port0 /dev/serial/by-path/pci-0000:00:14.0-usb-0:3.1:1.0-port0'
property[DEVNAME] = '/dev/ttyUSB0'
property[DEVPATH] = '/devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3.1/1-3.1:1.0/ttyUSB0/tty/ttyUSB0'
property[ID_AUTOSUSPEND] = '1'
property[ID_BUS] = 'usb'
property[ID_MODEL] = 'USB2.0-Ser_'
property[ID_MODEL_ENC] = 'USB2.0-Ser\x21'
property[ID_MODEL_FROM_DATABASE] = 'CH340 serial converter'
property[ID_MODEL_ID] = '7523'
property[ID_PATH] = 'pci-0000:00:14.0-usb-0:3.1:1.0'
property[ID_PATH_TAG] = 'pci-0000_00_14_0-usb-0_3_1_1_0'
property[ID_PCI_CLASS_FROM_DATABASE] = 'Serial bus controller'
property[ID_PCI_INTERFACE_FROM_DATABASE] = 'XHCI'
property[ID_PCI_SUBCLASS_FROM_DATABASE] = 'USB controller'
property[ID_REVISION] = '0254'
property[ID_SERIAL] = '1a86_USB2.0-Ser_'
property[ID_TYPE] = 'generic'
property[ID_USB_CLASS_FROM_DATABASE] = 'Vendor Specific Class'
property[ID_USB_DRIVER] = 'ch341'
property[ID_USB_INTERFACES] = ':ff0102:'
property[ID_USB_INTERFACE_NUM] = '00'
property[ID_VENDOR] = '1a86'
property[ID_VENDOR_ENC] = '1a86'
property[ID_VENDOR_FROM_DATABASE] = 'QinHeng Electronics'
property[ID_VENDOR_ID] = '1a86'
property[MAJOR] = '188'
property[MINOR] = '0'
property[SUBSYSTEM] = 'tty'
property[TAGS] = ':systemd:'
property[USEC_INITIALIZED] = '10677556659'
tag: systemd
sysattr[dev] = '188:0'
sysattr[device] = '(null)'
sysattr[power/autosuspend_delay_ms] = '(null)'
sysattr[power/control] = 'auto'
sysattr[power/runtime_active_time] = '0'
sysattr[power/runtime_status] = 'unsupported'
sysattr[power/runtime_suspended_time] = '0'
sysattr[subsystem] = 'tty'
sysattr[uevent] = 'MAJOR=188
MINOR=0
DEVNAME=ttyUSB0'

We can start by adding ID_VENDOR_FROM_DATABASE and ID_PATH; we can also additionally show ID_VENDOR and ID_MODEL if they are different from database ones (I think those are fetched by querying USB descriptors).

However, this (already) results in a quite long description string. We will experiment and see how to represent it better -- maybe, via a tooltip, or a multiline combobox item, or some other way.

Bottom line: I heard you; we will add extra information about serial devices on Linux in future IO Ninja releases.

Please try replacing the kernel module sources in share/tdevmon/src/lkm with these:

lkm-src.tar.xz

It looks like there's a bug in page protection removal code on ARM introduced in tdevmon-3.3.9...

Does it initially crash?

Please reboot your Raspberry Pi, build and load tdevmon.ko, then, instead of running IO Ninja, navigate to tdevmon-3.3.9-linux-arm32/bin and run:

$ ./tdevmon -m /dev/ttyUSB0

Will it segfault?