Dear friends!

It has been a long time since we released an update for IO Ninja -- more than eight months! Such a delay happened because we were working hard on the new major release, IO Ninja 5. And I'm happy to announce that it's finally out!

The most significant change from the previous generation of IO Ninja is the introduction of accounts, capabilities, and subscriptions. Please read the intro on our website to learn more about our motivation for such a major change. The dedicated section of the user manual covers this new ecosystem in more detail.

The IO Ninja 5 also features two new plugins dedicated to helping web developers debug modern WebSocket-based applications. Please read more about these plugins on the respective pages: WebSocket Client and WebSocket Server.

IO Ninja 5 includes a critical update of the Device Monitor Service for Linux that enables running the Serial Monitor on modern Linux kernels. There's also a major clean-up of UI for the macOS version, critical bug fixes and optimizations in the logging engine, and plenty of other important updates and improvements.

Besides the software updates, with IO Ninja 5, we introduce a new e-store that seamlessly integrates with the main website and provides a much more smooth experience. And of course, we -- finally! -- opened this forum, just as so many of you suggested. Here, you can ask questions, share your suggestions on improving IO Ninja, and receive support -- both from the IO Ninja Team and other users.

We worked with passion and dedication to release this new version of IO Ninja. We are sure that IO Ninja will help you in your work, and you will enjoy using it!

Currently, IAS (in-app-script) can only control the underlying session via:

• connect()
• disconnect()
• transmit(p, size)

Fine-tuning of the sessions configuration is currently not possible from IAS. But we plan to add this capability in the near future.

Each session class will export a public IAS interface (e.g., a serial session will have properties to query and control baud rate, parity, status lines, etc.) Then, IAS should be able to access this interface via a global constant g_session (or something like that).

Hello Bartosz,

The script for a filter to do what you want is very simple. First, you deduce the state of the CS line (from the I2cSpiTapLogRecordCode.SpiStart and SpiStop log records); then, you use this state to either hide or show the MOSI/MISO data (the log.StdRecordCode.TxRx log records).

The source code for such a filter might look something like this:

import "doc_Layer.jnc"
import "I2cSpiTap/I2cSpiTapLogRecordCode.jnc"

class SpiCsFilterLayer:
	doc.Layer,
	log.Filter
{
protected:
	ui.EnumProperty* m_csFilterProp; // a property to choose the filtering strategy
	int m_cs; // the state of the CS line

public:
	construct(doc.PluginHost* pluginHost);

	override bool filter(
		uint64_t timestamp,
		uint64_t recordCode,
		void const* p,
		size_t size
		);
}

SpiCsFilterLayer.construct(doc.PluginHost* pluginHost)
{
	basetype.construct(pluginHost);

	ui.EnumPropertyOption csFilterOptions[] = {
		{ "Show always", -1 },
		{ "Show when CS low", 0 },
		{ "Show when CS high", 1 },
	}

	m_csFilterProp = m_pluginHost.m_propertyGrid.createEnumProperty(
		"MOSI/MISO filter",
		"Show MOSI/MISO filtering criteria",
		csFilterOptions,
		countof(csFilterOptions)
	);

	m_cs = -1; // -1 means unknown
	pluginHost.m_log.addFilter(this);
}

bool SpiCsFilterLayer.filter(
	uint64_t timestamp,
	uint64_t recordCode,
	void const* p,
	size_t size
	)
{
	bool isVisible = true;

	switch (recordCode)
	{
	case log.StdRecordCode.SessionStarted:
		m_cs = -1; // reset to unknown
		break;

	case I2cSpiTapLogRecordCode.SpiStart:
		m_cs = 0;
		break;

	case I2cSpiTapLogRecordCode.SpiStop:
		m_cs = 1;
		break;

	case log.StdRecordCode.TxRx:
		isVisible =
			m_csFilterProp.m_value == -1 || // show always
			m_csFilterProp.m_value == m_cs; // matches the current state of CS
		break;
	}

	return isVisible;
}

An archive with the complete filter plugin is attached (SpiCsFilter.7z)

Usage:

1. Start I2C/SPI session
2. Open the SpiCsFilter.njplg file to attach this filter
3. Go to "Settings"
4. Select the appropriate filtering criteria
5. Hit "Apply and rebuild log" to apply the filter to the whole log (clicking "Apply" or "OK" will apply the filter "from now on" (i.e., to the follow-up log records only)

A little bit late for a follow-up, but still Which OS are you running on? Just FYI, ioninja-5.2.0 now picks up the system-wide dark theme on macOS and Linux with KDE desktops. If you running one of those, you can try the newly added dark theme support io IO Ninja. If not, stay tuned, as we do plan to put some extra effort and allow switching IO Ninja to dark mode at the user's will.

Hello,

Apologies for the delayed response. Support for njlog output directly from ioninja-hwc is on our TODO list (it was already requested by users). For the time being, you need to post-process the raw output of ioninja-hwc using any scripting language of your choice.

The protocol of communication between ioninja-hwc and plugins such as Serial over SSH or Serial Tap over SSH is not currently documented, but the decoding process and all relevant constants and data structures can be looked up in the sources of those plugins (remember, all IO Ninja plugins are open-sourced and available at ioninja/scripts/plugins).

In particular, all the relevant constants and packet structures for the ioninja-hwc protocol are contained in ioninja/scripts/common/io_HwcProto.jnc. The output file generated by ioninja-hwc is basically a sequence of "out" messages of this protocol. Each message starts with HwcMsgHdr followed by extra data block, the meaning of which depends on the message code. For example, HwcMsgCode.Rx is followed by the received bytes, HwcMsgCode.SerialTapCtsDsr is followed by HwcSerialStatusLines flags, etc.

Try decoding the output of ioninja-hwc using any language of your choice, and feel free to let me know if you run into a stumbling block or have any other questions.

Thanks for the feedback!

Auto-baud rate could be convenient, but when used in a sniffer, there's a quirk -- detecting baud rate can't be 100% reliable without prior knowledge of the incoming data stream. Serial protocols that support auto-baud rate always use specific packet headers that allow reliable detection of the baud rate for this exact reason. So, a sniffer with auto-baud rate detection (which can't make any assumptions about the underlying data) could initially produce incorrect bytes before it deduces the actual baud rate.

All that said, yes, we do consider adding the automatic baud rate detection in the new generation of our Serial Tap devices.

Not exactly a bug but rather a missing misuse protection.

So, what's happening? The automatic calculation of Modbus TCP frames is done like this:

void updateModbusTcpLength(void* p) {
	size_t size = dynamic sizeof(p);
	if (size < sizeof(io.ModbusTcpAduHdr))
		return;

	((io.ModbusTcpAduHdr*)p).m_length = size - offsetof(io.ModbusTcpAduHdr.m_deviceAddress);
}

Basically, m_length is set to the length of the payload (from m_deviceAddress and to the very end of the packet).

In your case, the packet length is 17, offset of m_deviceAddress is 6, so m_length is set to 11. However, the size of Modbus TCP read should always be 12 (not 17). If you erase the trailing 5 bytes from your packet, it will work as expected.

That should solve the issue; for more relevant details, read on.

By design, packet templates never allow creating a packet shorter than the template itself. However, it's allowed to add any suffix after the templated headers (that would be required for Modbus TCP write, for instance).

In ioninja-5.1.0 we added the fixedSize attribute exactly to prevent unintentional misuse in case of packets that should never grow longer than the template itself.

Modbus TCP read is exactly this kind of packet, but in the current version of IO Ninja, it's missing the fixedSize attribute. We will add it in the next release of IO Ninja, but in the meantime, you can fix it by yourself. To do so, please open ioninja/scripts/packets/ModbusTcp.jnc in any text editor, locate the definition of ModbuTcpReadPacket (in the very beginning), and modify it as such:

[ 
	displayName = "Modbus TCP read",
	fixedSize
]
struct ModbusTcpReadPacket {
// ...

In order to apply it, please reload this template into the Packet Template editor (Edit packet template -> Load stock script -> Modbus TCP).

Let me know if this works for you.

Confirmed. There's a regression in the Ethernet Tap plugin on Linux builds that somehow got under the radar during routine pre-release testing; will be fixed in the very next release.

Workarounds:

1. use ioninja-hwc to capture to .pcap from command line:

$ ./ioninja-hwc --ethernet-tap --pcap --out=my-capture.pcap

2. use the previous release of ioninja: https://tibbo.com/downloads/archive/ioninja/ioninja-5.5.1/

3. if that's possible in your case, use the windows or macos builds of the latest ioninja-5.6.0 (the regression only affects Linux builds).

Also, is it mandatory to be root, or is there a way to allow a user to open the Ethernet tap ?

You can add a UDEV rule to assign less restrictive permissions to USB devices based on VID/PID:

https://ioninja.com/doc/kb/linux_usb_permissions.html

For the Ethernet Tap, use these parameters:

SUBSYSTEM=="usb", ATTR{idVendor}=="326f", ATTR{idProduct}=="0003", MODE="0666", SYMLINK="ethernet-tap"

With IO Ninja, you can monitor Modbus TCP, analyze communications, and export data for further processing.

However, there's no out-of-the-box integration with data acquisition services.

If it's the monitor-and-analyze part you are interested in, then the roadmap is as follows.

The first stage is to acquire the TCP conversation between Modbus Master and Modbus Slave. In IO Ninja, this can be done via two alternative approaches.

(a) You can capture Ethernet packets using Ethernet Tap or Pcap Sniffer (capturing via Pcap could require some trickery if the Modbus Master is not running on the same workstation as IO Ninja), then reconstruct TCP conversation by attaching TCP Flow Analyzer.

(b) But a more straightforward approach would be to use IO Ninja as a man-in-the-middle and let it forward TCP traffic between the Master and Slave using TCP Proxy. Whenever this man-in-the-middle approach is applicable (i.e., you are in control of the Master and can redirect it to IO Ninja), that should be your choice -- it's much more straightforward and reliable.

After you set everything up for reconstruction of the raw TCP conversation between Master and Slave, you attach the Modbus Analyzer to decode the raw bytes into human-readable Modbus frames.

IO Ninja log is very powerful and should be more than enough to inspect and analyze the Modbus conversation.

If you need to post-process it further, one option would be to create a Log Layer to do the job (remember, IO Ninja is scriptable!). However, for people who have never touched IO Ninja scripting before, a much more straightforward path would be to save the log as .njlog and use any scripting language of their choice to walk over it.

The .njlog file format is very simple and all the relevant data structures & constants are open-source and defined in scripts/api/log_RecordCode.jnc and scripts/api/log_RecordFile.jnc (included in IO Ninja packages).

Hope this helps; feel free to follow up with any questions!

Hello Ivan,

capture serial communication with up to 3.3mbps

3.3 Mbps is beyond the maximum baud rate supported by Serial Tap. Baud rates <= 1 Mbps are guaranteed to be reliably captured; ~2 Mbps could work in theory, but errors are possible, and I think 3 Mbps would be outright rejected by the UART controller we use in Serial Tap.

add custom protocol parsing logic

This is possible; IO Ninja supports so called "layer" plugins that can transform original logs and add decoded protocol messages. See the Modbus Analyzer for an example of such layer plugin (it's open-source and located at /scripts/plugins/Modbus/)

perform time event measurement

Events are timestamped on the PC side -- the Serial Tap reports raw bytes and status line changes, IO Ninja adds timestamps and writes them to log.

be controlled by Python with ability to get parsed data, set time markers ad get time for each events

If you need to implement custom processing in Python, you can use ioninja-hwc (https://ioninja.com/features/hardware-client.html#cli) -- capture Serial Tap events from the command line and redirect those to a file or stdout; then implement whichever custom logic you need in Python.

Unfortunately, I can't see the wiring or the mode switch position from the photo. However, on the screenshot, all the data is RX (green). Most likely, it's the usual two-wire RS485 (half-duplex), but the Modbus Analyzer is configured for full-duplex (TX Master, RX Slave) -- which is causing those parsing errors.

We have a KB article on configuring Modbus for half-duplex links:
https://ioninja.com/doc/kb/modbus_half_duplex.html

Please let me know if this works for you.

@gary-biagioni

I guess the above just goes into the scripting panes. One problem I have is how do I debug the program I write. Not sure how to be able to do that, for example, look at the contents of a variable.

You can use the good old printf debugging.

import "hexEncoding.jnc"

void main() {
	int a = 10;
	char s[] = "abcdef";
	char buf[] = 0x"01 02 03 04 05";
	
	string_t msg = $"a: %1/0x%(1;02x) s: %2 buf: %3"(a, s, encodeHexString(buf, sizeof(buf), ' '));
	// string_t msg = $"a: %d/0x%02x s: %s buf: %s"(a, a, s, encodeHexString(buf, sizeof(buf), ' ')); // same
	printf($"This will go to the system log: $msg\n");
	g_logWriter.write(log.StdRecordCode.PlainText, $"This will go to the normal log: $msg\n");
}

The system log can be viewed via Menu->View->System Log

@gary-biagioni

That said above this looks super interesting to handle responses

https://ioninja.com/doc/developer-manual/tutorial-ias-server.html

At the moment of writing this tutoiral, onLogRecord was the only approach to reading incoming data bytes.

Of course, it still works, but we now have receive and receiveAll functions. I believe they are much easier to use.

Hi Gary,

Sorry, the API documentation is currently just a placeholder; it's auto-generated from the script sources (you can also check those sources directly at $IONINA_DIR/scripts/api/). However, we also have some scripting tutorials in the Developer Manual (https://ioninja.com/doc/developer-manual/tutorials.html) -- those should be helpful.

For the in-app-scripting inside the "Script" pane, all the available function declarations can be found at scripts/api/ias.jnc

To answer your question, the receive and receiveAll functions are used to collect raw data bytes that appear in your log as the RX (incoming) stream.

receive is the most basic method; it accepts the buffer and timeout and works as such:

1. if there are any incoming data bytes at the moment of call -- those data bytes are returned immediately;

2. otherwise, receive waits for incoming data for at most timeout milliseconds before returning. If a chunk of incoming data arrives before timeout expires, it will be returned as soon as received. If timeout expires and no data arrives -- 0 will be returned. timeout defaults to -1 which is equivalent to infinite wait for data (receive will not return until at least one byte of incoming data arrives).

receiveAll is a helper wrapper around receive and is used to fill the supplied buffer entirely (you can see its implementation in ias.jnc). This is convenient when your script needs to fill a fixed-size packet header before proceeding. If a packet arrives in chunks, the receiveAll will wait until the buffer is complete and only then return. The overloaded version of receiveAll with the timeout parameter will attempt to fill the buffer entirely -- but will bail after timeout milliseconds.

Hope this helps; feel free to ask more.

Are those little icons on the snapshot indicates the type of connection besides it is a pipe? (Socket, shared memory, message queue)

Pipe Monitor shows named and anonymous pipes only.

Anonymous pipe opens will be marked as (unnamed). If you run a Win32 code to create anonymous pipes such as:

	HANDLE hReadPipe;
	HANDLE hWritePipe;
	dword_t actualSize;
	char data[] = "abcdefghi";
	char buffer[1024];

	::CreatePipe(&hReadPipe, &hWritePipe, NULL, 0);

	::WriteFile(hWritePipe, data, sizeof(data), &actualSize, NULL);
	::ReadFile(hReadPipe, buffer, sizeof(buffer), &actualSize, NULL);

	::CloseHandle(hReadPipe);
	::CloseHandle(hWritePipe);

You should see something like:

The screenshot failed to upload; could you try again, please?

Regarding the IPC method, well, the Pipe Monitor shows communications over named and anonymous pipes. To see if a particular read or write is issued over an anonymous or named pipe (and the name of the pipe), you have to follow the log up all the way to the pipe open operation for this particular file ID; there you'll see the file name and the role (client or server).

Yes, you can use Serial Tap to monitor RS422 communications. Use the RS485 portion of the terminal block to connect your TX+/- and RX+/- lines.

Hello,

ModbusRtuWritePacket is from the legacy Modbus packet template library; it doesn't support multiple values per packet. The new Modbus plugin uses structures defined in scripts/protocols/io_Modbus.jnc. If you need to do it programmatically, just assemble your packet structure from necessary chunks (ADU hdr, PDU hdr, function-specific params, values, CRC). Important -- be sure to add pragma(Alignment, 1) as to avoid unintended struct paddings!

After the packet structure is defined, fill in the fields and calculate the checksum. Modbus RTU checksum is CRC16 ANSI (with init 0xffff) of the whole frame excluding the checksum itself.

The full script listing is below:

import "io_Modbus.jnc"

void main() {
	enum {
		DeviceAddress   = 1,
		RegisterAddress = 0xA000,
		RegisterCount   = 2,
	}

	pragma(Alignment, 1)
	struct MyPacket:
		io.ModbusRtuAduHdr,
		io.ModbusPduHdr,
		io.ModbusWriteMultipleParams {

		bigendian uint16_t m_registers[RegisterCount];
		uint16_t m_crc;
	}

	MyPacket packet;
	packet.m_deviceAddress = DeviceAddress;
	packet.m_func = io.ModbusFunc.WriteMultipleRegisters;
	packet.m_address = RegisterAddress;
	packet.m_count = RegisterCount;
	packet.m_size = sizeof(packet.m_registers);
	packet.m_registers[0] = 123;
	packet.m_registers[1] = 456;
	// ...
	packet.m_crc = crc16_ansi(packet, offsetof(packet.m_crc), -1);

	transmit(packet, sizeof(packet));
}