Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.
Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).
Hello Kent,
Apologies for not getting back to you sooner. Looks like you figured it out by yourself already
See my answer to your questions below.
I would like to know how I can get rid of the "SS High" messages. I had to leave them in so that the individual SPI frames would not be all joined together as one big blob of data.
To prevent merging of the last log record with the upcoming ones, add a "break" like this:
writer.write(timestamp, log.StdRecordCode.Break);
I assume I need to use a more powerful tool than filter if I want delete the message, but cause a the packet to be emitted. I assume that's a "converter".
A converter is more powerful, yes -- as it can also add extra records (or transform existing ones) while a filter can only hide or show original records.
What is the format of the data for the TxRx packets.
The data size is always even; the first half is MOSI, and the second is MISO.
Does the data change when the "swap" button is active, or is that done by a UI filter.
Pressing the button flips the first and second half in upcoming TxRx records. This is just a convenience tool, so that if you accidentally flipped MOSI and MISO when connecting the Tap, you can use this button instead of re-wiring.
TxRx
How does one signal to the generic TxRx formatter that a packet has ended and to emit current buffer.
Records are always written to the log immediately, i.e., when you call writer.write(...) or return false from the log.Converter.convert(...) method.
writer.write(...)
false
log.Converter.convert(...)
I assume, the question is how to prevent merging of Tx, Rx, or TxRx records. Then you do this by inserting a Break record where appropriate (e.g., instead of SpiStart records).
Tx
Rx
Break
SpiStart
Are the messages and data formats passed by the "I2C / SPI Tap" documented? That would be most helpful.
They are not documented in a dedicated reference manual, but the code that defines all the related records and converts them into the visual representation you see on-screen is open-source.
Please see scripts/plugins/I2cSpiTap/I2cSpiTapLogRecordCode.jnc, scripts/plugins/I2cSpiTap/I2cSpiTapLogRepresenter.jnc
scripts/plugins/I2cSpiTap/I2cSpiTapLogRecordCode.jnc
scripts/plugins/I2cSpiTap/I2cSpiTapLogRepresenter.jnc
FRAMING error happens when the stop bit(s) is not set where it should be (so the byte frame was not correctly received). Unless it only happens occasionally (and I mean, almost-never kind of occasionally), it's a sign to double-check serial settings (baud rate, data bits, parity bit, stop bits) and wiring (make sure to minimize possible electrical disturbances).
Hello Jose,
Thank you so much for a great report!
It took me some time to check and test everything; please read my comments below.
You expect pcap dumps captured with ioninja-hwc and with tcpdump to be exactly the same. Actually, that's not guaranteed. Ethernet Tap does not "pause" forwarding of Ethernet frames from one port to the other if the Cypress USB buffer is full -- that's in line with how other taps for IO Ninja work (they don't even "forward" data and are simply connected to the existing lines instead). The buffer on the Ethernet Tap itself is very small (as it has no RAM/flash), so if the software is not fast enough to read the captured data -- data will be lost).
ioninja-hwc
tcpdump
The packet loss ratio you experienced with ioninja-hwc is, of course, not acceptable. The original version of ioninja-hwc was doing the initiating of USB read transfers, parsing the input from the Tap and writing it to the output file (a pipe to Wireshark, in your case) all in one thread. So if somewhere along the way any delay happens, then new USB reads are not scheduled, the Cypress USB buffer on the Ethernet Tap gets full, and the packets are lost.
I rewrote ioninja-hwc so that scheduling of USB reads is now done in a dedicated thread, so any blocking waits during writing to the output file/pipe should not prevent new USB reads from being scheduled. Also, it's now directly controllable how many simultaneous USB read transfers are passed to the bus driver (as well as buffer sizes for those).
Please try this build:
https://tibbo.com/downloads/archive/ioninja/.internal/ioninja-hwc-test.tar.xz
Controlling the buffer sizes and read parellelism is done via --read-parallelism and --read-block-size command line parameters. It also generates a log of overflows in %HOME%/ioninja-hwc-log.txt
--read-parallelism
--read-block-size
%HOME%/ioninja-hwc-log.txt
Still, this can't completely prevent packet loss -- Linux/Windows/macOS are not realtime OSes, and at 10+ MBytes per second, any delay could result in an overflow of the (rather tiny) buffer on the Ethernet Tap. Considering this, it's recommended to connect Ethernet Tap to a fast host to minimize packet loss.
The ultimate solution should include some RAM/SD/eMMC on the Ethernet Tap to store captured packets.
Besides packet loss, there's a bigger problem.
Under some conditions during heavy load, the FPGA firmware could send garbaged data over USB (not only the packet contents, but the USB protocol headers could suffer as well). Luckily -- or, maybe, unfortunately (because it essentially hid the problem) -- our USB protocol is designed in a way so that it recovers and re-synchronizes almost instantly, so this problem remained unnoticed. But yes, I was able to reproduce the issue. Data corruption over a bulk USB endpoint is not acceptable and should never happen. Our FPGA developer is investigating the issue at the moment. Hopefully, we will have a firmware patch for this soon.
Once again, thank you for reporting the issue in great detail; I'll keep you posted.
Hello again Jose,
Thanks for your feedback on our taps! I also believe that using the USB for sniffing instead of a NIC (even a virtual one, i.e., when plugging a tap essentially creates a dedicated NIC for sniffing) is a more convenient and overall better approach.
Regarding issues you described above -- please try the latest build for RPi:
https://tibbo.com/downloads/archive/ioninja/ioninja-5.3.2/ioninja-5.3.2-b-linux-arm32.tar.xz https://tibbo.com/downloads/archive/ioninja/ioninja-5.3.2/ioninja-5.3.2-b-linux-arm64.tar.xz
All of those should be fixed (please do let me know if not).
The issues are confirmed and (mostly) fixed already. I will try to make a service release this week. I can also make an RPi build for you to try first.
Does it show traffic over other pipes? If it does, it might be a filtering issue; when applying a capture or a display filter, don't prefix it with \\.\pipe\
\\.\pipe\
The most easy-to-remember way of writing a filename filter is to simply start it with a *, for example: *discord-ipc-*
*
*discord-ipc-*
If the issue persists, please provide a screenshot or a .njlog file.
.njlog
Both issues confirmed.
The Homebrew repo was not updated regarding the new file format with the CPU suffix. Fixed.
The arm64 build of IO Ninja used QT from Homebrew -- which, apparently, is either built with some weird rpath settings, or patched after installation (didn't dig too deep, but otool shows apparent problems). Fixed by rebuilding QT for arm64 and using this build in IO Ninja (v5.3.2-a)
arm64
rpath
otool
v5.3.2-a
Please try again and let me know if it works for you now.
What does uname -a say?
uname -a
My first guess would be that you are running a 32-bit Linux on RPi4; if that's the case, you need to grab an arm32 package of tdevmon.
arm32
tdevmon
The issue is confirmed.
Apparently, in linux-6.2.0, they introduced a new check that prevents copying from iov_iter unless it's an iterator created for a write operation. Hence, the read_iter hook failed to copy data from the buffer.
linux-6.2.0
iov_iter
read_iter
Please try the latest build tdevmon-3.3.12 (available at the Downloads page) and let me know if it works for you.
tdevmon-3.3.12
Hello Mick,
You have access to all the IO Ninja capabilities, and Replay Log doesn't require IO Ninja hardware.
First of all, let's make sure there are no misunderstandings about what Replay Log is capable of and which scenarios it can be used in.
After capturing a log of communications, the "Replay Log" plugin can send previously captured TX or RX packets with respect to timestamps, i.e. transmit them again inserting the same delays between packets as observed during the original comm session. In order to do that, the target session (the one you attach Replay Log to) should be writeable. You can't replay a log over a monitor session such as Serial Monitor or Serial Tap, simply because those sessions are read-only and can't transmit. This is why the "Replay" button was grayed out.
However, you can first capture a log using a Serial Monitor session, and then replay it using a Serial session. For example, there is a device attached to a serial port on your machine. Capture all the input from this device using Serial Monitor, then replace this serial device with another USB-to-Serial adapter and open this second adapter from a Serial session. This second Serial session can be used to transmit packets in lieu of your device. Now you can attach the Replay Log layer and replay the log previously captured by Serial Monitor, thus repeating all the commands originally sent by your device.
Hope this clarifies the matter!
Hi again, Chaim,
From the first screenshot, I assume, you are running on Windows, correct? Then I will use Windows-style paths and declarations in the samples below.
Yes. You can open the necessary plugin(s) and start capturing from the command line. There are multiple ways of achieving that, and the easiest would be this.
First, configure the session the way you need -- i.e., start the main plugin (e.g., Serial Tap), attach required analyzer plugins (e.g., Modbus RTU, or your own analyzer), and configure all the necessary settings using the UI. Then, save the session somewhere (e.g., C:/Projects/playground/ioninja/my-analyzer-session/my-analyzer-session.njssn)
C:/Projects/playground/ioninja/my-analyzer-session/my-analyzer-session.njssn
Now, you can start IO Ninja passing it a path to .njssn to automatically open this session. To automatically start capturing, also pass -c (i.e., connect):
.njssn
-c
ioninja -c <path-to-njssn>
Yes. One of Jancy design goals was easy and efficient interoperability with C/C++ code. Again, multiple ways of calling external code exist, the most straightforward would be declaring an interface to your C/C++ library via the dynamiclib construct.
dynamiclib
Let's say, you have a library called my_lib.dll with the following exported C function:
my_lib.dll
extern "C" __declspec(dllexport) size_t my_inverse( void* out, size_t outSize, const void* in, size_t inSize ) { // inverse the in-buffer and write it to out-buffer }
In your Jancy code, declare an interface to this library like this:
pragma(ThinPointers, true); dynamiclib MyLib { size_t my_inverse( void* out, size_t outSize, const void* in, size_t inSize ); } pragma(ThinPointers, default);
Basically, function declarations are the same as in C/C++; you only have to make sure to use "thin" pointers.
Jancy pointers are "fat" -- they include extra information about region, type, etc. To achieve ABI compatibility with C/C++, we need to explicitly declare pointers as "thin", like this:
char thin* p;
If there could be many FFI declarations -- like in an interface to a C/C++ library -- it may be more convenient to implicitly treat ALL pointers as "thin" using pragma(ThinPointers, true).
pragma(ThinPointers, true)
Now, when you need to call your library, load and call it like this:
MyLib myLib; myLib.open("path-to-your-lib/my_lib.dll"); myLib.lib.my_inverse(p1, size1, p2, size2);
Hope this helps!
Yes, you can use TX/RX Filter to hide the data flowing in either one of the directions.
A new internal build of IO Ninja for Aarch64 is up:
https://tibbo.com/downloads/archive/ioninja/.internal/prerelease/ioninja-5.3.1-b-linux-arm64.deb https://tibbo.com/downloads/archive/ioninja/.internal/prerelease/ioninja-5.3.1-b-linux-arm64.tar.xz
Tried it on the latest ArchLinux -- it's working now.
Let me know if you notice any issues.
Documentation in general needs some serious improvements, that's true Working on that.
In the meantime, we try our best to provide developer-level support and assistance -- here or via the support email.
Yes, it's possible; you can control all parameters of a serial port (including the break condition) from your script via the "dispatch" object g_session:
g_session
void main() { connect(); // configure the port g_session.m_baudRate = 38400; g_session.m_dataBits = 8; // ... for (;;) { g_session.m_break = !g_session.m_break; // toggle break condition sys.sleep(1000); } }
Note: there's a typo in the serial dispatch script in the latest official release ioninja-5.3.1 (will be fixed in the very next service release). In the meantime, for the script above to work, please open to scripts/plugins/Serial/SerialDispatch.jnc and replace:
ioninja-5.3.1
scripts/plugins/Serial/SerialDispatch.jnc
SerialDispatchCode.GetBreak -> SerialDispatchCode.GetBreakCondition and SerialDispatchCode.SetBreak -> SerialDispatchCode.SetBreakCondition
SerialDispatchCode.GetBreak
SerialDispatchCode.GetBreakCondition
SerialDispatchCode.SetBreak
SerialDispatchCode.SetBreakCondition
Let me know if this works for you.
tdevmon should now build and work on new Aarch64 kernels -- with or without huge page support (ArchLinux kernel is built with it, Raspbian -- without).
Please try it and let me know:
https://tibbo.com/downloads/archive/tdevmon/tdevmon-3.3.11/tdevmon-3.3.11-linux-arm64.tar.xz
Re ioninja-hwc -- I wouldn't expect issues on Aarch64; it's a simple console app that does no platform-specific trickery.
Hmm. Everything is working well on the fully updated Raspbian Aarch64; I installed ArchLinux ARM Aarch64 -- and everything fails there the same way you describe. Fixing the compilation error for tdevmon was easy, but that didn't quite cut it -- it looks like write protection removal needs extra polishing for the newer kernels.
No prob: log_RepresentStruct.jnc
Just FYI, here's the added part (special handling for struct fields):
/// ... if (field.m_type.m_typeKind == jnc.TypeKind.Struct) { representation.addHyperText($"%1%2:\t"(indent.m_sz, displayName)); representStruct( representation, dynamic (jnc.StructType*)field.m_type, displayName,, p + field.m_offset, baseOffset + field.m_offset, 0, 0, // no [+/-] header, always expanded indentLevel + 1 ); continue; } /// ...
Having sub-fields collapsible is theoretically possible but won't scale well -- we have 32 total fold-flags per record, so it's better to assign those statically (rather than from a loop in a recursive function). But I think having struct fields always expanded should be good enough in most cases.
https://tibbo.com/downloads/archive/ioninja/.internal/ioninja-5.3.1-a-linux-arm64.deb https://tibbo.com/downloads/archive/ioninja/.internal/ioninja-5.3.1-a-linux-arm64.tar.xz https://tibbo.com/downloads/archive/tdevmon/.internal/tdevmon-3.3.11-linux-arm64.tar.xz
Let me know if these work for you!
Hello Chaim,
Glad to hear you've made some good progress with your protocol analyzer!
Yes, log.representStruct should simplify and at the same time "standardize" representation of binary-based packets. And yes, you are right -- at the moment, it doesn't recursively represent all the nested structs. That's because this functionality was never required in the stock protocol analyzers in IO Ninja -- we use "layered" representation rather than "nested" because (a) this makes resulting tables more compact and easier to navigate, and (b) allows us to colorize layers using different colors. But there's nothing that prevents adding the support for nesting to log.representStruct; I think we can add that to the upcoming release.
log.representStruct
How deep is the nesting in your case? Could you please provide definitions of the packet structs in your protocol?
Just FYI, the implementation of log.representStruct is in scripts/common/log_RepresentStruct.jnc; you can try experimenting with it by yourself until the official release.
scripts/common/log_RepresentStruct.jnc