Posts made by Vladimir

Hello Kent,

Apologies for not getting back to you sooner. Looks like you figured it out by yourself already

See my answer to your questions below.

I would like to know how I can get rid of the "SS High" messages. I had to leave them in so that the individual SPI frames would not be all joined together as one big blob of data.

To prevent merging of the last log record with the upcoming ones, add a "break" like this:

writer.write(timestamp, log.StdRecordCode.Break);

I assume I need to use a more powerful tool than filter if I want delete the message, but cause a the packet to be emitted. I assume that's a "converter".

A converter is more powerful, yes -- as it can also add extra records (or transform existing ones) while a filter can only hide or show original records.

What is the format of the data for the TxRx packets.

The data size is always even; the first half is MOSI, and the second is MISO.

Does the data change when the "swap" button is active, or is that done by a UI filter.

Pressing the button flips the first and second half in upcoming TxRx records. This is just a convenience tool, so that if you accidentally flipped MOSI and MISO when connecting the Tap, you can use this button instead of re-wiring.

How does one signal to the generic TxRx formatter that a packet has ended and to emit current buffer.

Records are always written to the log immediately, i.e., when you call writer.write(...) or return false from the log.Converter.convert(...) method.

I assume, the question is how to prevent merging of Tx, Rx, or TxRx records. Then you do this by inserting a Break record where appropriate (e.g., instead of SpiStart records).

Are the messages and data formats passed by the "I2C / SPI Tap" documented? That would be most helpful.

They are not documented in a dedicated reference manual, but the code that defines all the related records and converts them into the visual representation you see on-screen is open-source.

Please see scripts/plugins/I2cSpiTap/I2cSpiTapLogRecordCode.jnc, scripts/plugins/I2cSpiTap/I2cSpiTapLogRepresenter.jnc

FRAMING error happens when the stop bit(s) is not set where it should be (so the byte frame was not correctly received). Unless it only happens occasionally (and I mean, almost-never kind of occasionally), it's a sign to double-check serial settings (baud rate, data bits, parity bit, stop bits) and wiring (make sure to minimize possible electrical disturbances).

Hello Jose,

Thank you so much for a great report!

It took me some time to check and test everything; please read my comments below.

You expect pcap dumps captured with ioninja-hwc and with tcpdump to be exactly the same. Actually, that's not guaranteed. Ethernet Tap does not "pause" forwarding of Ethernet frames from one port to the other if the Cypress USB buffer is full -- that's in line with how other taps for IO Ninja work (they don't even "forward" data and are simply connected to the existing lines instead). The buffer on the Ethernet Tap itself is very small (as it has no RAM/flash), so if the software is not fast enough to read the captured data -- data will be lost).

The packet loss ratio you experienced with ioninja-hwc is, of course, not acceptable. The original version of ioninja-hwc was doing the initiating of USB read transfers, parsing the input from the Tap and writing it to the output file (a pipe to Wireshark, in your case) all in one thread. So if somewhere along the way any delay happens, then new USB reads are not scheduled, the Cypress USB buffer on the Ethernet Tap gets full, and the packets are lost.

I rewrote ioninja-hwc so that scheduling of USB reads is now done in a dedicated thread, so any blocking waits during writing to the output file/pipe should not prevent new USB reads from being scheduled. Also, it's now directly controllable how many simultaneous USB read transfers are passed to the bus driver (as well as buffer sizes for those).

Please try this build:

https://tibbo.com/downloads/archive/ioninja/.internal/ioninja-hwc-test.tar.xz

Controlling the buffer sizes and read parellelism is done via --read-parallelism and --read-block-size command line parameters. It also generates a log of overflows in %HOME%/ioninja-hwc-log.txt

Still, this can't completely prevent packet loss -- Linux/Windows/macOS are not realtime OSes, and at 10+ MBytes per second, any delay could result in an overflow of the (rather tiny) buffer on the Ethernet Tap. Considering this, it's recommended to connect Ethernet Tap to a fast host to minimize packet loss.

The ultimate solution should include some RAM/SD/eMMC on the Ethernet Tap to store captured packets.

Besides packet loss, there's a bigger problem.

Under some conditions during heavy load, the FPGA firmware could send garbaged data over USB (not only the packet contents, but the USB protocol headers could suffer as well). Luckily -- or, maybe, unfortunately (because it essentially hid the problem) -- our USB protocol is designed in a way so that it recovers and re-synchronizes almost instantly, so this problem remained unnoticed. But yes, I was able to reproduce the issue. Data corruption over a bulk USB endpoint is not acceptable and should never happen. Our FPGA developer is investigating the issue at the moment. Hopefully, we will have a firmware patch for this soon.

Once again, thank you for reporting the issue in great detail; I'll keep you posted.

Hello again Jose,

Thanks for your feedback on our taps! I also believe that using the USB for sniffing instead of a NIC (even a virtual one, i.e., when plugging a tap essentially creates a dedicated NIC for sniffing) is a more convenient and overall better approach.

Regarding issues you described above -- please try the latest build for RPi:

https://tibbo.com/downloads/archive/ioninja/ioninja-5.3.2/ioninja-5.3.2-b-linux-arm32.tar.xz
https://tibbo.com/downloads/archive/ioninja/ioninja-5.3.2/ioninja-5.3.2-b-linux-arm64.tar.xz

All of those should be fixed (please do let me know if not).

Hello Jose,

The issues are confirmed and (mostly) fixed already. I will try to make a service release this week. I can also make an RPi build for you to try first.

Does it show traffic over other pipes? If it does, it might be a filtering issue; when applying a capture or a display filter, don't prefix it with \\.\pipe\

The most easy-to-remember way of writing a filename filter is to simply start it with a *, for example: *discord-ipc-*

If the issue persists, please provide a screenshot or a .njlog file.

Both issues confirmed.

1. The Homebrew repo was not updated regarding the new file format with the CPU suffix. Fixed.

2. The arm64 build of IO Ninja used QT from Homebrew -- which, apparently, is either built with some weird rpath settings, or patched after installation (didn't dig too deep, but otool shows apparent problems). Fixed by rebuilding QT for arm64 and using this build in IO Ninja (v5.3.2-a)

Please try again and let me know if it works for you now.

What does uname -a say?

My first guess would be that you are running a 32-bit Linux on RPi4; if that's the case, you need to grab an arm32 package of tdevmon.

The issue is confirmed.

Apparently, in linux-6.2.0, they introduced a new check that prevents copying from iov_iter unless it's an iterator created for a write operation. Hence, the read_iter hook failed to copy data from the buffer.

Please try the latest build tdevmon-3.3.12 (available at the Downloads page) and let me know if it works for you.

Hello Mick,

You have access to all the IO Ninja capabilities, and Replay Log doesn't require IO Ninja hardware.

First of all, let's make sure there are no misunderstandings about what Replay Log is capable of and which scenarios it can be used in.

After capturing a log of communications, the "Replay Log" plugin can send previously captured TX or RX packets with respect to timestamps, i.e. transmit them again inserting the same delays between packets as observed during the original comm session. In order to do that, the target session (the one you attach Replay Log to) should be writeable. You can't replay a log over a monitor session such as Serial Monitor or Serial Tap, simply because those sessions are read-only and can't transmit. This is why the "Replay" button was grayed out.

However, you can first capture a log using a Serial Monitor session, and then replay it using a Serial session. For example, there is a device attached to a serial port on your machine. Capture all the input from this device using Serial Monitor, then replace this serial device with another USB-to-Serial adapter and open this second adapter from a Serial session. This second Serial session can be used to transmit packets in lieu of your device. Now you can attach the Replay Log layer and replay the log previously captured by Serial Monitor, thus repeating all the commands originally sent by your device.

Hope this clarifies the matter!

Hi again, Chaim,

From the first screenshot, I assume, you are running on Windows, correct? Then I will use Windows-style paths and declarations in the samples below.

1. Command line

Yes. You can open the necessary plugin(s) and start capturing from the command line. There are multiple ways of achieving that, and the easiest would be this.

First, configure the session the way you need -- i.e., start the main plugin (e.g., Serial Tap), attach required analyzer plugins (e.g., Modbus RTU, or your own analyzer), and configure all the necessary settings using the UI. Then, save the session somewhere (e.g., C:/Projects/playground/ioninja/my-analyzer-session/my-analyzer-session.njssn)

Now, you can start IO Ninja passing it a path to .njssn to automatically open this session. To automatically start capturing, also pass -c (i.e., connect):

ioninja -c <path-to-njssn>

2. FFI (Foreign Function Interface)

Yes. One of Jancy design goals was easy and efficient interoperability with C/C++ code. Again, multiple ways of calling external code exist, the most straightforward would be declaring an interface to your C/C++ library via the dynamiclib construct.

Let's say, you have a library called my_lib.dll with the following exported C function:

extern "C"
__declspec(dllexport) 
size_t 
my_inverse(
	void* out,
	size_t outSize,
	const void* in,
	size_t inSize
) {
	// inverse the in-buffer and write it to out-buffer
}

In your Jancy code, declare an interface to this library like this:

pragma(ThinPointers, true);

dynamiclib MyLib {
	size_t 
	my_inverse(
		void* out,
		size_t outSize,
		const void* in,
		size_t inSize
	);
}

pragma(ThinPointers, default);

Basically, function declarations are the same as in C/C++; you only have to make sure to use "thin" pointers.

Jancy pointers are "fat" -- they include extra information about region, type, etc. To achieve ABI compatibility with C/C++, we need to explicitly declare pointers as "thin", like this:

char thin* p;

If there could be many FFI declarations -- like in an interface to a C/C++ library -- it may be more convenient to implicitly treat ALL pointers as "thin" using pragma(ThinPointers, true).

Now, when you need to call your library, load and call it like this:

MyLib myLib;
myLib.open("path-to-your-lib/my_lib.dll");
myLib.lib.my_inverse(p1, size1, p2, size2);

3. Creating an external process is possible, too, of course. However, passing binary data to a process via the command line -- and especially getting an answer back -- would be much less convenient and efficient. If you need to perform some data processing in external code, I strongly recommend using the method shown above, i.e., create a C/C++ library and call it from Jancy. It's both easy to implement and as efficient as it gets in runtime.

Hope this helps!

Yes, you can use TX/RX Filter to hide the data flowing in either one of the directions.

A new internal build of IO Ninja for Aarch64 is up:

https://tibbo.com/downloads/archive/ioninja/.internal/prerelease/ioninja-5.3.1-b-linux-arm64.deb
https://tibbo.com/downloads/archive/ioninja/.internal/prerelease/ioninja-5.3.1-b-linux-arm64.tar.xz

Tried it on the latest ArchLinux -- it's working now.

Let me know if you notice any issues.

Documentation in general needs some serious improvements, that's true Working on that.

In the meantime, we try our best to provide developer-level support and assistance -- here or via the support email.

Yes, it's possible; you can control all parameters of a serial port (including the break condition) from your script via the "dispatch" object g_session:

void main() {
	connect();

	// configure the port
	g_session.m_baudRate = 38400;
	g_session.m_dataBits = 8;
	// ...

	for (;;) {
		g_session.m_break = !g_session.m_break; // toggle break condition
		sys.sleep(1000);
	}
}

Note: there's a typo in the serial dispatch script in the latest official release ioninja-5.3.1 (will be fixed in the very next service release). In the meantime, for the script above to work, please open to scripts/plugins/Serial/SerialDispatch.jnc and replace:

SerialDispatchCode.GetBreak -> SerialDispatchCode.GetBreakCondition
and
SerialDispatchCode.SetBreak -> SerialDispatchCode.SetBreakCondition

Let me know if this works for you.

tdevmon should now build and work on new Aarch64 kernels -- with or without huge page support (ArchLinux kernel is built with it, Raspbian -- without).

Please try it and let me know:

https://tibbo.com/downloads/archive/tdevmon/tdevmon-3.3.11/tdevmon-3.3.11-linux-arm64.tar.xz

Re ioninja-hwc -- I wouldn't expect issues on Aarch64; it's a simple console app that does no platform-specific trickery.

Hmm. Everything is working well on the fully updated Raspbian Aarch64; I installed ArchLinux ARM Aarch64 -- and everything fails there the same way you describe. Fixing the compilation error for tdevmon was easy, but that didn't quite cut it -- it looks like write protection removal needs extra polishing for the newer kernels.

No prob: log_RepresentStruct.jnc

Just FYI, here's the added part (special handling for struct fields):

/// ...
		if (field.m_type.m_typeKind == jnc.TypeKind.Struct) {
			representation.addHyperText($"%1%2:\t"(indent.m_sz, displayName));
			representStruct(
				representation,
				dynamic (jnc.StructType*)field.m_type,
				displayName,,
				p + field.m_offset,
				baseOffset + field.m_offset,
				0, 0, // no [+/-] header, always expanded
				indentLevel + 1
			);

			continue;
		}
/// ...

Having sub-fields collapsible is theoretically possible but won't scale well -- we have 32 total fold-flags per record, so it's better to assign those statically (rather than from a loop in a recursive function). But I think having struct fields always expanded should be good enough in most cases.

https://tibbo.com/downloads/archive/ioninja/.internal/ioninja-5.3.1-a-linux-arm64.deb
https://tibbo.com/downloads/archive/ioninja/.internal/ioninja-5.3.1-a-linux-arm64.tar.xz
https://tibbo.com/downloads/archive/tdevmon/.internal/tdevmon-3.3.11-linux-arm64.tar.xz

Let me know if these work for you!

Hello Chaim,

Glad to hear you've made some good progress with your protocol analyzer!

Yes, log.representStruct should simplify and at the same time "standardize" representation of binary-based packets. And yes, you are right -- at the moment, it doesn't recursively represent all the nested structs. That's because this functionality was never required in the stock protocol analyzers in IO Ninja -- we use "layered" representation rather than "nested" because (a) this makes resulting tables more compact and easier to navigate, and (b) allows us to colorize layers using different colors. But there's nothing that prevents adding the support for nesting to log.representStruct; I think we can add that to the upcoming release.

How deep is the nesting in your case? Could you please provide definitions of the packet structs in your protocol?

Just FYI, the implementation of log.representStruct is in scripts/common/log_RepresentStruct.jnc; you can try experimenting with it by yourself until the official release.