Dear friends!

It has been a long time since we released an update for IO Ninja -- more than eight months! Such a delay happened because we were working hard on the new major release, IO Ninja 5. And I'm happy to announce that it's finally out!

The most significant change from the previous generation of IO Ninja is the introduction of accounts, capabilities, and subscriptions. Please read the intro on our website to learn more about our motivation for such a major change. The dedicated section of the user manual covers this new ecosystem in more detail.

The IO Ninja 5 also features two new plugins dedicated to helping web developers debug modern WebSocket-based applications. Please read more about these plugins on the respective pages: WebSocket Client and WebSocket Server.

IO Ninja 5 includes a critical update of the Device Monitor Service for Linux that enables running the Serial Monitor on modern Linux kernels. There's also a major clean-up of UI for the macOS version, critical bug fixes and optimizations in the logging engine, and plenty of other important updates and improvements.

Besides the software updates, with IO Ninja 5, we introduce a new e-store that seamlessly integrates with the main website and provides a much more smooth experience. And of course, we -- finally! -- opened this forum, just as so many of you suggested. Here, you can ask questions, share your suggestions on improving IO Ninja, and receive support -- both from the IO Ninja Team and other users.

We worked with passion and dedication to release this new version of IO Ninja. We are sure that IO Ninja will help you in your work, and you will enjoy using it!

Currently, IAS (in-app-script) can only control the underlying session via:

• connect()
• disconnect()
• transmit(p, size)

Fine-tuning of the sessions configuration is currently not possible from IAS. But we plan to add this capability in the near future.

Each session class will export a public IAS interface (e.g., a serial session will have properties to query and control baud rate, parity, status lines, etc.) Then, IAS should be able to access this interface via a global constant g_session (or something like that).

Hello Bartosz,

The script for a filter to do what you want is very simple. First, you deduce the state of the CS line (from the I2cSpiTapLogRecordCode.SpiStart and SpiStop log records); then, you use this state to either hide or show the MOSI/MISO data (the log.StdRecordCode.TxRx log records).

The source code for such a filter might look something like this:

import "doc_Layer.jnc"
import "I2cSpiTap/I2cSpiTapLogRecordCode.jnc"

class SpiCsFilterLayer:
	doc.Layer,
	log.Filter
{
protected:
	ui.EnumProperty* m_csFilterProp; // a property to choose the filtering strategy
	int m_cs; // the state of the CS line

public:
	construct(doc.PluginHost* pluginHost);

	override bool filter(
		uint64_t timestamp,
		uint64_t recordCode,
		void const* p,
		size_t size
		);
}

SpiCsFilterLayer.construct(doc.PluginHost* pluginHost)
{
	basetype.construct(pluginHost);

	ui.EnumPropertyOption csFilterOptions[] = {
		{ "Show always", -1 },
		{ "Show when CS low", 0 },
		{ "Show when CS high", 1 },
	}

	m_csFilterProp = m_pluginHost.m_propertyGrid.createEnumProperty(
		"MOSI/MISO filter",
		"Show MOSI/MISO filtering criteria",
		csFilterOptions,
		countof(csFilterOptions)
	);

	m_cs = -1; // -1 means unknown
	pluginHost.m_log.addFilter(this);
}

bool SpiCsFilterLayer.filter(
	uint64_t timestamp,
	uint64_t recordCode,
	void const* p,
	size_t size
	)
{
	bool isVisible = true;

	switch (recordCode)
	{
	case log.StdRecordCode.SessionStarted:
		m_cs = -1; // reset to unknown
		break;

	case I2cSpiTapLogRecordCode.SpiStart:
		m_cs = 0;
		break;

	case I2cSpiTapLogRecordCode.SpiStop:
		m_cs = 1;
		break;

	case log.StdRecordCode.TxRx:
		isVisible =
			m_csFilterProp.m_value == -1 || // show always
			m_csFilterProp.m_value == m_cs; // matches the current state of CS
		break;
	}

	return isVisible;
}

An archive with the complete filter plugin is attached (SpiCsFilter.7z)

Usage:

1. Start I2C/SPI session
2. Open the SpiCsFilter.njplg file to attach this filter
3. Go to "Settings"
4. Select the appropriate filtering criteria
5. Hit "Apply and rebuild log" to apply the filter to the whole log (clicking "Apply" or "OK" will apply the filter "from now on" (i.e., to the follow-up log records only)

A little bit late for a follow-up, but still Which OS are you running on? Just FYI, ioninja-5.2.0 now picks up the system-wide dark theme on macOS and Linux with KDE desktops. If you running one of those, you can try the newly added dark theme support io IO Ninja. If not, stay tuned, as we do plan to put some extra effort and allow switching IO Ninja to dark mode at the user's will.

Hello,

Apologies for the delayed response. Support for njlog output directly from ioninja-hwc is on our TODO list (it was already requested by users). For the time being, you need to post-process the raw output of ioninja-hwc using any scripting language of your choice.

The protocol of communication between ioninja-hwc and plugins such as Serial over SSH or Serial Tap over SSH is not currently documented, but the decoding process and all relevant constants and data structures can be looked up in the sources of those plugins (remember, all IO Ninja plugins are open-sourced and available at ioninja/scripts/plugins).

In particular, all the relevant constants and packet structures for the ioninja-hwc protocol are contained in ioninja/scripts/common/io_HwcProto.jnc. The output file generated by ioninja-hwc is basically a sequence of "out" messages of this protocol. Each message starts with HwcMsgHdr followed by extra data block, the meaning of which depends on the message code. For example, HwcMsgCode.Rx is followed by the received bytes, HwcMsgCode.SerialTapCtsDsr is followed by HwcSerialStatusLines flags, etc.

Try decoding the output of ioninja-hwc using any language of your choice, and feel free to let me know if you run into a stumbling block or have any other questions.

Thanks for the feedback!

Auto-baud rate could be convenient, but when used in a sniffer, there's a quirk -- detecting baud rate can't be 100% reliable without prior knowledge of the incoming data stream. Serial protocols that support auto-baud rate always use specific packet headers that allow reliable detection of the baud rate for this exact reason. So, a sniffer with auto-baud rate detection (which can't make any assumptions about the underlying data) could initially produce incorrect bytes before it deduces the actual baud rate.

All that said, yes, we do consider adding the automatic baud rate detection in the new generation of our Serial Tap devices.

Not exactly a bug but rather a missing misuse protection.

So, what's happening? The automatic calculation of Modbus TCP frames is done like this:

void updateModbusTcpLength(void* p) {
	size_t size = dynamic sizeof(p);
	if (size < sizeof(io.ModbusTcpAduHdr))
		return;

	((io.ModbusTcpAduHdr*)p).m_length = size - offsetof(io.ModbusTcpAduHdr.m_deviceAddress);
}

Basically, m_length is set to the length of the payload (from m_deviceAddress and to the very end of the packet).

In your case, the packet length is 17, offset of m_deviceAddress is 6, so m_length is set to 11. However, the size of Modbus TCP read should always be 12 (not 17). If you erase the trailing 5 bytes from your packet, it will work as expected.

That should solve the issue; for more relevant details, read on.

By design, packet templates never allow creating a packet shorter than the template itself. However, it's allowed to add any suffix after the templated headers (that would be required for Modbus TCP write, for instance).

In ioninja-5.1.0 we added the fixedSize attribute exactly to prevent unintentional misuse in case of packets that should never grow longer than the template itself.

Modbus TCP read is exactly this kind of packet, but in the current version of IO Ninja, it's missing the fixedSize attribute. We will add it in the next release of IO Ninja, but in the meantime, you can fix it by yourself. To do so, please open ioninja/scripts/packets/ModbusTcp.jnc in any text editor, locate the definition of ModbuTcpReadPacket (in the very beginning), and modify it as such:

[ 
	displayName = "Modbus TCP read",
	fixedSize
]
struct ModbusTcpReadPacket {
// ...

In order to apply it, please reload this template into the Packet Template editor (Edit packet template -> Load stock script -> Modbus TCP).

Let me know if this works for you.

Confirmed. There's a regression in the Ethernet Tap plugin on Linux builds that somehow got under the radar during routine pre-release testing; will be fixed in the very next release.

Workarounds:

1. use ioninja-hwc to capture to .pcap from command line:

$ ./ioninja-hwc --ethernet-tap --pcap --out=my-capture.pcap

2. use the previous release of ioninja: https://tibbo.com/downloads/archive/ioninja/ioninja-5.5.1/

3. if that's possible in your case, use the windows or macos builds of the latest ioninja-5.6.0 (the regression only affects Linux builds).

Also, is it mandatory to be root, or is there a way to allow a user to open the Ethernet tap ?

You can add a UDEV rule to assign less restrictive permissions to USB devices based on VID/PID:

https://ioninja.com/doc/kb/linux_usb_permissions.html

For the Ethernet Tap, use these parameters:

SUBSYSTEM=="usb", ATTR{idVendor}=="326f", ATTR{idProduct}=="0003", MODE="0666", SYMLINK="ethernet-tap"

No worries; I'm happy to hear that the issue is resolved now!

If you tried the Latin-1 and still saw unexpected coloring, it could have been the case sensitivity issue -- this has caught me off guard a few times as well.

Regarding why we use UTF-8 in regex by default. UTF-8 is the default encoding across IO Ninja (log engine, terminal, transmit pane, etc). So it makes sense to use UTF-8 in regex for the sake of consistency. But here we have a dilemma. If the regex engine uses UTF-8 by default, then individual bytes could be uncolored -- RE2 could treat them as part of a UTF-8 sequence. If, on the other hand, we use Latin-1 by default, then multi-byte Unicode characters could be uncolored.

I guess we could try to be smarter and automatically choose Latin-1 or UTF-8 based on the pattern (i.e., force Latin-1 when the pattern contains \xHH, force UTF-8 when the pattern contains multi-byte Unicode characters). Forcing the encoding is not a good thing, though. Maybe have an "Auto" option or something like that.

If you encounter anything else, please let me know. Your feedback over the years has been invaluable—thank you so much!

Also, regarding this:

and, in a weird case, a value (0x77) colorize two bytes (0x77 and 0x57)

This is actually fine. "Case sensitive" is set to OFF, so 0x77 (W) and 0x57 (w) both match.

Hello Josep,

You have to set "Force Latin-1 encoding" when you are colorizing raw byte sequences. Otherwise, RE2 will try to decode UTF-8, and this yields unexpected results when the data stream (or the pattern) contains invalid UTF-8 sequences -- which is a common thing in raw IO streams.

As a matter of fact, I think we should change the default behaviour -- i.e., use Latin-1 by default and only UTF-8-decode when explicitly asked for.

Currently, there's no built-in UI feature for that. However, it's a totally legit feature request (and is quite easy to implement). We could add it to one of the upcoming releases.

At the moment, what can be done instead is a simple plugin to expand ALL records:

class AutoExpandFilterLayer:
	doc.Layer,
	log.FoldingFilter {

	ui.BoolProperty* m_expandProp;

	construct(doc.PluginHost* pluginHost) {
		basetype.construct(pluginHost);
		m_expandProp = pluginHost.m_propertyGrid.createBoolProperty("Expand all records");
		pluginHost.m_log.addFoldingFilter(this);
	}

	override void restoreDefaultProperties() {
		m_expandProp.m_value = true;
	}

	override uint8_t filter(
		uint64_t timestamp,
		uint64_t recordCode,
		void const* p,
		size_t size
	) {
		return m_expandProp.m_value ? log.FoldFlags.ExpandAll : 0;
	}
}

Here's an archive with the full plugin:

AutoExpand.7z

After attaching it, all the new foldable records will be expanded by default. You can check/uncheck the "Expand all" property and rebuild the log to expand or collapse all foldable records.

Hope this helps!

This script only generates packet contents and passes the raw data to the underlying session for transmission -- so it will work with any transport. If you need to send those packets over UDP, open the "UDP Socket" plugin, configure remote IP:Port accordingly, then run the script:

Here's a script that will open a CSV file, parse it line by line, then prepare and transmit binary packets for each line:

import "io_base.jncx"
import "io_MappedFile.jnc"

enum: uint64_t {
	// epoch difference (in seconds) between Unix time (1 Jan 1970 00:00) and Windows time (1 Jan. 1601 00:00)
	UnixTimeEpochDiff = 11644473600,

	// delay between packets (in milliseconds)
	InterPacketDelay  = 500, 
}

// the structure of the packet

pragma(Alignment, 1)

struct Packet {
	uint8_t m_optionsByte             = 0x83;
	uint8_t m_mobileIdLength          = 0x08;
	bigendian uint64_t m_mobileId;
	uint8_t m_mobileIdTypeLen         = 0x01;
	uint8_t m_mobileIdType            = 0x02;
	uint8_t m_serviceType             = 0x01;
	uint8_t m_messageType             = 0x02;
	bigendian uint16_t m_sequenceIdx;
	bigendian uint32_t m_updateTime;
	bigendian uint32_t m_timeOfFix;
	bigendian uint32_t m_latitude;
	bigendian uint32_t m_longitude;
	bigendian uint32_t m_altitude;
	bigendian uint32_t m_speed;
	bigendian uint16_t m_heading;
	uint8_t m_satellites              = 0x0B;
	uint8_t m_fixStatus               = 0x02;
	bigendian uint16_t m_carrier      = 0x0004;
	bigendian uint16_t m_rssi         = 0xFFBF;
	uint8_t m_commState               = 0x0F;
	uint8_t m_hdop                    = 0x09;
	uint8_t m_inputs;
	uint8_t m_unitStatus              = 0x01;
	uint8_t m_eventIndex              = 0x04;
	uint8_t m_eventCode               = 0xA8;
	uint8_t m_accums                  = 0x06;
	uint8_t m_spare                   = 0x00;
	bigendian uint32_t m_accum0       = 0x00000000;
	bigendian uint32_t m_accum1       = 0x00000000;
	bigendian uint32_t m_accum2       = 0x10600000;
	bigendian uint32_t m_accum3       = 0x001A35DF;
	bigendian uint32_t m_accum4       = 0x02C80269;
	bigendian uint32_t m_accum5       = 0x62347EF0;
	char m_lf                         = '\n';
}

char const* findEol(
	char const* p,
	char const* eof
) {
	char const* eol = memchr(p, '\n', eof - p);
	return eol ? eol + 1 : eof;
}

void main() {
	connect();

	string_t fileName = io.getHomeDir() + "/history.csv"; // adjust accordingly
	io.MappedFile file;
	file.open(fileName, io.FileOpenFlags.ReadOnly);

	size_t size = file.m_size;
	char const* p = file.view(0, size);
	char const* eof = p + size;
	size_t index = 0;

	Packet packet; // all const fields are initialized; we'll adjust variable fields below

	p = findEol(p, eof); // skip the first line
	while (p < eof) { // process the rest line by line	
		char const* eol = findEol(p, eof);
		string_t line(p, eol - p);
		p = eol;
		
		if (line !~ r"\s*([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)")
			continue; // skip all CSV lines that do not match the pattern

		// Jancy uses Windows timestamps (in 100-nsec intervals, i.e. sec / 10^7)
		uint_t unixTime = sys.getTimestamp() / 10000000 - UnixTimeEpochDiff;		

		// adjust all the variable fields in the packet
		packet.m_mobileId = strtoul($1,, 16);
		packet.m_sequenceIdx = index++;
		packet.m_updateTime = unixTime;
		packet.m_timeOfFix = unixTime;
		packet.m_latitude = (uint_t)(atof($3) * 10000000);
		packet.m_longitude = (uint_t)(atof($4) * 10000000);
		packet.m_altitude = (uint_t)(atof($5) * 100);
		packet.m_speed = (uint_t)(atof($6) * 100);
		packet.m_heading = strtoul($7);
		packet.m_inputs = strtoul($8);

		// transmit, wait and move on to the next line
		transmit(&packet, sizeof(packet));
		sys.sleep(InterPacketDelay); 
	}
}

Unlike the original PHP sample I received from you, I didn't generate a HEX string first -- in Jancy, it's much cleaner (and more efficient, of course) to generate binary data right away. First, we declare the Packet struct according to your specification and initialize all constant fields; then, we walk over the file line by line and adjust fields that depend on the CSV data.

Feel free to ask any questions regarding the script.

Confirmed. There's a regression in the Ethernet Tap plugin on Linux builds that somehow got under the radar during routine pre-release testing; will be fixed in the very next release.

Workarounds:

1. use ioninja-hwc to capture to .pcap from command line:

$ ./ioninja-hwc --ethernet-tap --pcap --out=my-capture.pcap

2. use the previous release of ioninja: https://tibbo.com/downloads/archive/ioninja/ioninja-5.5.1/

3. if that's possible in your case, use the windows or macos builds of the latest ioninja-5.6.0 (the regression only affects Linux builds).

Also, is it mandatory to be root, or is there a way to allow a user to open the Ethernet tap ?

You can add a UDEV rule to assign less restrictive permissions to USB devices based on VID/PID:

https://ioninja.com/doc/kb/linux_usb_permissions.html

For the Ethernet Tap, use these parameters:

SUBSYSTEM=="usb", ATTR{idVendor}=="326f", ATTR{idProduct}=="0003", MODE="0666", SYMLINK="ethernet-tap"

@apie-llc

I can see Serial Tap under Device Manager USB controllers-> IO Ninja Serial Tap, location Port_#0001.Hub_#0003
When I start a new session for ModBus Analyzer, the port list just empty.

Which session are you running? I mean, Modbus Analyzer is a layer plugin that's attached on top of some other session that provides raw data. I assume, you want to capture data via a Serial Tap -- then you have to start a Serial Tap session first:

Then you can attach Modbus Analyzer on top so that it decodes Modbus commands & replies for you.

The devmon is installed and works properly as in the KB section

devmon is for capturing local traffic generated by other apps on the same PC; it's not required for a Serial Tap.

If you have another application doing Modbus communications, you can start a Serial Monitor session (this plugin uses devmon) and see what this other app sends and receives. Then you can attach a Modbus layer on top to parse raw data and decode Modbus packets.

@jose-marro

I was thinking of using it for parsing and generating frames (IEC60870-5-103 frames)...

For parsing frames, dylayout would be a perfect tool.

For generating frames programmatically, dynamic layouts won't add much extra convenience (and dyfields are currently all const anyway).

But! One thing I didn't mention above is that dynamic layouts are also perfect for creating packet templates!

After defining a dynamic layout "specification" for a protocol, you will be able to conveniently build packets for this protocol in the Property Grid on the Hex Transmit pane in IO Ninja. Set enumeration fields via drop-down lists, set bits in bitfields with check-boxes, have big-endian automatically converted for you, etc. If a protocol uses checksums, you can define methods for automatic calculation of those checksums before transmission.

All in all, it's an awesome tool for generating and sending out test packets! If you didn't see it, please check it out -- it could be just what you are looking for.

A short follow-up after thinking more about q1.

In my previous write-up, I used the word "packets" when talking about binary blobs that dynamic layouts work with. But of course, those could be any binary objects — disk files, disks themselves, shared memory, etc.

Indeed, with packets, we usually generate the whole thing from scratch—and using dynamic layouts here doesn't offer much.

But if we think about objects like files or disks, it makes perfect sense to allow modification alongside parsing. Something like (1) locate a specific field inside a file, (2) modify this field, (3) proceed to the next one.

So yeah, I think we should remove the forced const on dyfield declarations. We still need to somehow preserve const-correctness for the parse-only cases, though.

One way would be to introduce an auxiliary class jnc.MutableDynamicLayout, which would take non-const pointers; when the dylayout argument is jnc.MutableDynamicLayout, the Jancy compiler won't add implicit const to dyfield declarations.

Thoughts?

P.S. Moved the topic to General Discussion