Posts made by Vladimir

Here's an internal build of IO Ninja for arm64:

https://tibbo.com/downloads/archive/ioninja/.internal/prerelease/ioninja-5.7.1-c-linux-arm64.tar.xz

The ioninja-hwc introduces two new parameters:

--split-size <bytes> and --split-time <seconds>

Both options can be used together; in this case, the splitting will occur based on whichever condition is met first. Both options support reasonable suffixes, e.g., --split-size=10M, --split-time=6h, etc.

Let me know if this works for you!

If this is something you could add "for free" to your new development, great! If not, well, I think that this is no a common case, in fact, I found it very weird (but "real" indeed).

Yes, it comes for free, so we'll have a setting for that!

if possible/feasible/reasonable, consider the options of splitting by size (eg. each "n" MB) and by time (eg. each "t" minutes). It is just an idea... perhaps it seems more logic just consider the size, but -eg. for my case- it is more convenient to have captures of info more or less time aligned (eg. daily aligned... in fact, I am currently splitting them each 6 hours)

Good point! Let's have settings for both!

consider the option of being "transparent" when power-off...

Oh yes, I agree! This is absolutely one of the must-have features for the next generation of Ethernet Tap. Disconnecting power from the tap should not disrupt the existing link.

Well, then have to add help tooltips to every other setting, too, for consistency... Which could be the right thing to do, actually

In this particular case, however, I'm kinda inclined to simplify and drop this setting altogether (use hex-line-size / 2 for this special dual-hex-view mode).

Hello,

The Serial Tap for IO Ninja (and the majority of standard USB-to-Serial adapters) does not support 9-bit UART communications. The upcoming Serial Tap Pro (to be released early next year) will support this feature natively, but in the meantime, here’s a workaround.

First, some background. The most common use case for the 9th bit in UART is marking the address byte in some industrial automation protocols. This important optimization enables slave nodes to inspect only the first (address) byte; if the address does not match, the slave can safely ignore all subsequent bytes until the next address byte (i.e., the next UART frame with the 9th bit set).

In these scenarios, the 9th bit is clear for most payload bytes and set only for address bytes. To monitor such communications using a Serial Tap (or a USB-to-RS485 adapter), use the settings as follows:

Baud rate: <specify-the-correct-baud-rate>
Data bits: 8
Parity: Space
Stop bits: 1

With these settings, address bytes will generate PARITY errors (the parity is set to "space," but the parity bit is 1), while the rest of the traffic will remain free of line errors.

Reporting of line errors is inherently asynchronous, so the PARITY errors won't necessarily align precisely with each address byte (BTW, Serial Tap Pro will address this, too, and line errors will be byte-precise). Still, you should see a PARITY error somewhere around the address byte.

Hope this helps; let me know if this workaround works for you!

Hello,

It's not very clear what you mean by "automatically labeling log files according to the session name or device ID". When you create a new IO Ninja session, the log is stored in a temporary file. When you save the session, you explicitly specify a directory name to store the session data (or a single file name if you only need to save the log file) -- so you are the one picking the file names.

If you need to post-process the log files externally, you can do it in any language of your choice (Python, JavaScript, C, etc.) The .njlog file format is very simple. A brief overview can be found here: https://ioninja.com/doc/developer-manual/logging-engine.html) and all the internal structures are open-source; see scripts/api/log_RecordCode.jnc and scripts/api/log_RecordFile.jnc for all the relevant declarations.

Let me know if you have any follow-up questions regarding the file format.

Apologies for the inconvenience.

Please restart the Pipe Monitor session and set the Filter to "None" if you don't need to filter by name. Alternatively, you can set the name wildcard to * (i.e., an asterisk).

Tech details:

The latest release of IO Ninja contains a regression that results in an exception when setting an empty wildcard as a filter; the log engine remains suspended after this exception, thus resulting in the "empty log" you saw.

Here's how to fix it (until the next release takes care of it):

1. Open <ioninja-dir>/scripts/common/log_MonitorFilter.jnc in any text editor
2. Search for MonitorFilter.setFilter
3. Add the following snippet right before processing the wildcard string:

switch (filterKind) {
case MonitorFilterKind.FileName:
case MonitorFilterKind.ProcessName:
	// <<<<<<<
	if (!filter) {
		m_filterKind = MonitorFilterKind.None;
		break;
	}
	// >>>>>>>

Save the script and restart the Pipe Monitor session (no need to restart the app).

Please let me know if it works for you.

This was done for the I2C/SPI Tap. The IO Ninja log engine supports this special dual-hex view for SPI which looks like this:

But I totally see how this could be confusing -- it's only relevant to this single plugin and a single mode in this plugin!

Maybe, we should remove this setting completely and simply calculate it as hex-line-size / 2?

Hello again, Josep,

Yes, it's in the works, but it's currently sidelined in favor of the upcoming release of Serial Tap Pro -- which I mentioned in my response to your post about the TTL signal inversion.

Full-scale development of the new generation of Ethernet Tap will be resumed after that.

Hello again, Josep,

Unfortunately, it's not feasible to add such a feature to the existing Serial Tap (which is based on a fixed-function dual-channel USB-to-TTL module from Cypress that doesn't support it natively).

But we can add this feature of level inversion for TTL to the upcoming FPGA-based Serial Tap Pro. It's about to be released very soon, actually (in a few months!) and will have many new features such as galvanic isolation of signals, high baud rates (up to 2Mbps), support for 9-bit UART, per-byte high-precision timestamps (1mcs), guarantee of correct sequencing for all events (TX/RX, status line changes, parity & framing errors, etc.), data injection, and more.

We're excited about this release, which will be a huge step forward compared to the current generation of Serial Tap!

Hello Josep,

I apologize once again for the delayed response

You’ve made a great point! The ability to auto-split the output file into volumes could indeed be very useful (and is also trivial to implement). We’ll include something like the --split command-line option in the next release of IO Ninja.

If you like, I can send you a link to an internal build of ioninja-hwc for arm64 once this feature is ready (before the official release). Let me know!

No worries; I'm happy to hear that the issue is resolved now!

If you tried the Latin-1 and still saw unexpected coloring, it could have been the case sensitivity issue -- this has caught me off guard a few times as well.

Regarding why we use UTF-8 in regex by default. UTF-8 is the default encoding across IO Ninja (log engine, terminal, transmit pane, etc). So it makes sense to use UTF-8 in regex for the sake of consistency. But here we have a dilemma. If the regex engine uses UTF-8 by default, then individual bytes could be uncolored -- RE2 could treat them as part of a UTF-8 sequence. If, on the other hand, we use Latin-1 by default, then multi-byte Unicode characters could be uncolored.

I guess we could try to be smarter and automatically choose Latin-1 or UTF-8 based on the pattern (i.e., force Latin-1 when the pattern contains \xHH, force UTF-8 when the pattern contains multi-byte Unicode characters). Forcing the encoding is not a good thing, though. Maybe have an "Auto" option or something like that.

If you encounter anything else, please let me know. Your feedback over the years has been invaluable—thank you so much!

Also, regarding this:

and, in a weird case, a value (0x77) colorize two bytes (0x77 and 0x57)

This is actually fine. "Case sensitive" is set to OFF, so 0x77 (W) and 0x57 (w) both match.

Hello Josep,

You have to set "Force Latin-1 encoding" when you are colorizing raw byte sequences. Otherwise, RE2 will try to decode UTF-8, and this yields unexpected results when the data stream (or the pattern) contains invalid UTF-8 sequences -- which is a common thing in raw IO streams.

As a matter of fact, I think we should change the default behaviour -- i.e., use Latin-1 by default and only UTF-8-decode when explicitly asked for.

Currently, there's no built-in UI feature for that. However, it's a totally legit feature request (and is quite easy to implement). We could add it to one of the upcoming releases.

At the moment, what can be done instead is a simple plugin to expand ALL records:

class AutoExpandFilterLayer:
	doc.Layer,
	log.FoldingFilter {

	ui.BoolProperty* m_expandProp;

	construct(doc.PluginHost* pluginHost) {
		basetype.construct(pluginHost);
		m_expandProp = pluginHost.m_propertyGrid.createBoolProperty("Expand all records");
		pluginHost.m_log.addFoldingFilter(this);
	}

	override void restoreDefaultProperties() {
		m_expandProp.m_value = true;
	}

	override uint8_t filter(
		uint64_t timestamp,
		uint64_t recordCode,
		void const* p,
		size_t size
	) {
		return m_expandProp.m_value ? log.FoldFlags.ExpandAll : 0;
	}
}

Here's an archive with the full plugin:

AutoExpand.7z

After attaching it, all the new foldable records will be expanded by default. You can check/uncheck the "Expand all" property and rebuild the log to expand or collapse all foldable records.

Hope this helps!

This script only generates packet contents and passes the raw data to the underlying session for transmission -- so it will work with any transport. If you need to send those packets over UDP, open the "UDP Socket" plugin, configure remote IP:Port accordingly, then run the script:

Here's a script that will open a CSV file, parse it line by line, then prepare and transmit binary packets for each line:

import "io_base.jncx"
import "io_MappedFile.jnc"

enum: uint64_t {
	// epoch difference (in seconds) between Unix time (1 Jan 1970 00:00) and Windows time (1 Jan. 1601 00:00)
	UnixTimeEpochDiff = 11644473600,

	// delay between packets (in milliseconds)
	InterPacketDelay  = 500, 
}

// the structure of the packet

pragma(Alignment, 1)

struct Packet {
	uint8_t m_optionsByte             = 0x83;
	uint8_t m_mobileIdLength          = 0x08;
	bigendian uint64_t m_mobileId;
	uint8_t m_mobileIdTypeLen         = 0x01;
	uint8_t m_mobileIdType            = 0x02;
	uint8_t m_serviceType             = 0x01;
	uint8_t m_messageType             = 0x02;
	bigendian uint16_t m_sequenceIdx;
	bigendian uint32_t m_updateTime;
	bigendian uint32_t m_timeOfFix;
	bigendian uint32_t m_latitude;
	bigendian uint32_t m_longitude;
	bigendian uint32_t m_altitude;
	bigendian uint32_t m_speed;
	bigendian uint16_t m_heading;
	uint8_t m_satellites              = 0x0B;
	uint8_t m_fixStatus               = 0x02;
	bigendian uint16_t m_carrier      = 0x0004;
	bigendian uint16_t m_rssi         = 0xFFBF;
	uint8_t m_commState               = 0x0F;
	uint8_t m_hdop                    = 0x09;
	uint8_t m_inputs;
	uint8_t m_unitStatus              = 0x01;
	uint8_t m_eventIndex              = 0x04;
	uint8_t m_eventCode               = 0xA8;
	uint8_t m_accums                  = 0x06;
	uint8_t m_spare                   = 0x00;
	bigendian uint32_t m_accum0       = 0x00000000;
	bigendian uint32_t m_accum1       = 0x00000000;
	bigendian uint32_t m_accum2       = 0x10600000;
	bigendian uint32_t m_accum3       = 0x001A35DF;
	bigendian uint32_t m_accum4       = 0x02C80269;
	bigendian uint32_t m_accum5       = 0x62347EF0;
	char m_lf                         = '\n';
}

char const* findEol(
	char const* p,
	char const* eof
) {
	char const* eol = memchr(p, '\n', eof - p);
	return eol ? eol + 1 : eof;
}

void main() {
	connect();

	string_t fileName = io.getHomeDir() + "/history.csv"; // adjust accordingly
	io.MappedFile file;
	file.open(fileName, io.FileOpenFlags.ReadOnly);

	size_t size = file.m_size;
	char const* p = file.view(0, size);
	char const* eof = p + size;
	size_t index = 0;

	Packet packet; // all const fields are initialized; we'll adjust variable fields below

	p = findEol(p, eof); // skip the first line
	while (p < eof) { // process the rest line by line	
		char const* eol = findEol(p, eof);
		string_t line(p, eol - p);
		p = eol;
		
		if (line !~ r"\s*([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)")
			continue; // skip all CSV lines that do not match the pattern

		// Jancy uses Windows timestamps (in 100-nsec intervals, i.e. sec / 10^7)
		uint_t unixTime = sys.getTimestamp() / 10000000 - UnixTimeEpochDiff;		

		// adjust all the variable fields in the packet
		packet.m_mobileId = strtoul($1,, 16);
		packet.m_sequenceIdx = index++;
		packet.m_updateTime = unixTime;
		packet.m_timeOfFix = unixTime;
		packet.m_latitude = (uint_t)(atof($3) * 10000000);
		packet.m_longitude = (uint_t)(atof($4) * 10000000);
		packet.m_altitude = (uint_t)(atof($5) * 100);
		packet.m_speed = (uint_t)(atof($6) * 100);
		packet.m_heading = strtoul($7);
		packet.m_inputs = strtoul($8);

		// transmit, wait and move on to the next line
		transmit(&packet, sizeof(packet));
		sys.sleep(InterPacketDelay); 
	}
}

Unlike the original PHP sample I received from you, I didn't generate a HEX string first -- in Jancy, it's much cleaner (and more efficient, of course) to generate binary data right away. First, we declare the Packet struct according to your specification and initialize all constant fields; then, we walk over the file line by line and adjust fields that depend on the CSV data.

Feel free to ask any questions regarding the script.

Confirmed. There's a regression in the Ethernet Tap plugin on Linux builds that somehow got under the radar during routine pre-release testing; will be fixed in the very next release.

Workarounds:

1. use ioninja-hwc to capture to .pcap from command line:

$ ./ioninja-hwc --ethernet-tap --pcap --out=my-capture.pcap

2. use the previous release of ioninja: https://tibbo.com/downloads/archive/ioninja/ioninja-5.5.1/

3. if that's possible in your case, use the windows or macos builds of the latest ioninja-5.6.0 (the regression only affects Linux builds).

Also, is it mandatory to be root, or is there a way to allow a user to open the Ethernet tap ?

You can add a UDEV rule to assign less restrictive permissions to USB devices based on VID/PID:

https://ioninja.com/doc/kb/linux_usb_permissions.html

For the Ethernet Tap, use these parameters:

SUBSYSTEM=="usb", ATTR{idVendor}=="326f", ATTR{idProduct}=="0003", MODE="0666", SYMLINK="ethernet-tap"

@apie-llc

I can see Serial Tap under Device Manager USB controllers-> IO Ninja Serial Tap, location Port_#0001.Hub_#0003
When I start a new session for ModBus Analyzer, the port list just empty.

Which session are you running? I mean, Modbus Analyzer is a layer plugin that's attached on top of some other session that provides raw data. I assume, you want to capture data via a Serial Tap -- then you have to start a Serial Tap session first:

Then you can attach Modbus Analyzer on top so that it decodes Modbus commands & replies for you.

The devmon is installed and works properly as in the KB section

devmon is for capturing local traffic generated by other apps on the same PC; it's not required for a Serial Tap.

If you have another application doing Modbus communications, you can start a Serial Monitor session (this plugin uses devmon) and see what this other app sends and receives. Then you can attach a Modbus layer on top to parse raw data and decode Modbus packets.