Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.
Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).
Hi Massimo!
Indeed, the filter is currently implemented in the session only -- it should be transferred to the log plugin instead; this way, it will be there when you open a standalone log. We definitely will fix this in the upcoming release (scheduled for May). In the meantime, here's a workaround:
Create a fresh Ethernet Tap session and save it (let's say, in $HOME/ethernet-tap-1). This will create 3 files under $HOME/ethernet-tap-1:
$HOME/ethernet-tap-1
ethernet-tap-1.njssn
ethernet-tap-1.njlog
ethernet-tap-1.njcfg
Close the session. Then, using any file manager, copy your original log file over ethernet-tap-1.njlog. Re-open this session, and it will use the original log file. Now filtering packets will be available -- together with all other actions of the Ethernet Tap session (e.g., capturing more data and appending it to the log).
Hope this works for you.
Once again, the issue with not being able to filter standalone Ethernet Tap/Pcap Sniffer logs will fixed in the next release.
I think it would be a very helpful feature to add to IO ninja to just be able to save a log as a raw binary file for each direction.
It's very easy to achieve with a log filter/observer plugin script.
However, I can't say I find it very useful to have two raw binary files containing just the raw bytes flowing in each directing -- how would you reconstruct the mapping between requests and replies in case of some non-trivial conversation between the client and server?
I often have serial port logs that I just want to save to a bin file as-is, and I end up closing IO ninja and just going back to something like CoolTerm which supports this natively.
If it's a uni-directional log kind of thing (e.g., your device keeps dumping debug printfs to a RS232, and you want to save it to a file) -- this can be done using Session Linking. Open a Serial session, open a File Stream session in Write-only mode and point it to some file, then "link" those sessions. Everything that the Serial session reads from the device, will be redirected to your file.
it also supports recording directly to a file, in case the computer crashes or powers off during logging
Actually, IO Ninja always keeps its logs in disk files. When you "Save Log", the temporary .njlog file simply gets copied into the file you select. But if you "Save Session", those .njlog files will get stored in the specified folder, and you will know exactly where to find them in case of a crash.
.njlog
But once again, a dedicated plugin that would explicitly save RX to one file and TX to another is quite easy to implement. I don't know, maybe we should add it to one of the upcoming releases as an official plugin (or a sample)?
Try the updated analyzer:
BacNetMsTp.7z
It checks CRCs now and discards frames with broken headers (in the previous log, the decoder didn't actually stop, but because a broken header specified a very long payload size, it kept buffering data assuming it still was a payload).
Also, when you upload the log, please detach the Analyzer before saving the log. This way, we can access the original raw & unprocessed data (instead of the already decoded frames).
No prob, I've increased the upload file size limit to 8MB.
Also, you could have archived it with 7z
7z
Apparently, this was a permission issue; it doesn't apply to administrators, so I didn't even know it was there...
Anyway, I've adjusted the file upload permissions for registered users; please try again.
Tried using the upload images(right most icon) to send a 7z zip file but I don't have privileges for it.
Not "Upload image", "Upload file". Does your "Reply toolbar" have this button?
How do I send you the log that is out of sync
Hit "Reply" (not "Post quick reply"); the rightmost button on the toolbar is "Upload File". 7-zip it first so that it has the .7z extension permitted by this forum.
.7z
Removing the padding gives this trying to load the Analyzer:
Most likely, you just commented out the padding code, but forgot to add a semicolon ; at the end of the statement. In Jancy, just like in C, declarations and expression statements must be terminated with a semicolon.
;
While using this analyzer should we have been running Settings->Log Engine->Binary Data->Binary Data Merge with a 3msec threshold.
This won't affect the analyzer, but I think it's completely unnecessary now. You will see packets boundaries anyway -- and not just boundaries, decoded header fields, too.
P.S. For simplicity, just sync on 0x55; that should be good enough.
0x55
In BacNetMsTpParser.parse(...), add this:
BacNetMsTpParser.parse(...)
while (p < end) { void const* p0 = p; // 0. sync on 0x55 if (!m_buffer.m_size) { void const* preamble = memchr(p, 0x55, end - p); if (!preamble) break; p = preamble; // skip everything before the preamble } ...
Version 5.5.0 will not display the contents of BACnet headers properly (after you click [+] to expand). The pre-release .7z file is a portable archive; extract it anywhere and run ioninja-5.5.1\bin\ioninja.exe.
ioninja-5.5.1\bin\ioninja.exe
Re "stop" of the analyzer -- could you share the original .njlog with the large packet?
I suspect that the reason is the incorrect handling of padding in the script (it went out of sync immediately after the large frame). Try removing the padding:
size_t frameSize = sizeof(BacNetMsTpHdr) + // header hdr.m_length + // payload sizeof(uint16_t); // CRC // (hdr.m_length & 1); // padding <---
Also, it makes sense to wait and sync on \x55\xff before starting buffering the packet; this way, the parser will re-sync after synchronization is lost (this still can occasionally happen because RS485 does not guarantee lossless delivery).
\x55\xff
P.S. FYI, just as with any representation generated by log.representStruct(), you can click on a header field, and the corresponding data bytes will be highlighted below -- so that you can see the mapping between header fields and data bytes.
log.representStruct()
How do you get to the Transmit pane if it is greyed out(un-selectable) ?
The Transmit pane is greyed out in the Serial Tap plugin because it's a read-only sniffer device. You can transmit using the Serial Terminal plugin and a regular USB-to-RS485 adapter.
I tried using RegEx Markup with default Session settings and it does not want to break the packets up.
After adding "Force Latin-1 encoding" to the Log Regex Markup settings I'm getting color highlights but no new timestamps when it see's the pattern.
Right, I should have mentioned that Latin-1 encoding might be necessary when the pattern contains invalid UTF-8 sequences (such as \x55\xff).
In general, using timestamps to split the data stream into packets is not a reliable enough approach (unless every captured byte is precisely timestamped). Serial Tap doesn't do that; instead, timestamps are assigned to the whole blocks of data as they are received over USB. There are multiple layers of buffering involved (at both the Serial Tap and the PC side), so the timing could be distorted to some extent, and bytes from different packets could end up being read within the same block. It's much better to parse the stream and split it into packets based on the actual data contents.
I've created a draft of the BACnet MSTP protocol analyzer for you: https://tibbo.com/downloads/archive/ioninja/.internal/scripts/BacNetMsTp.7z
It breaks the stream into BACnet frames, gives a human-readable representation of the frame header, and highlights the payload (if any).
Feel free to modify or extend it to your liking. However, please note, that ioninja-5.5.0 has a regression that prevents log.representStruct() used in this script from operating properly (it's already fixed but the service release is not out yet). So, to use this plugin, please either (a) roll back to ioninja-5.4.2 or (b) use the internal build ioninja-5.5.0-a: https://tibbo.com/downloads/archive/ioninja/.internal/prerelease/ioninja-5.5.0-a-windows-amd64.7z
ioninja-5.5.0
ioninja-5.4.2
ioninja-5.5.0-a
Let me know if it works for you, and feel free to ask me anything about the internal implementation of the analyzer script.
Hello John,
I'm really sorry for not answering your questions earlier; somehow this topic went under my radar...
The very first question (garbage input) you actually solved by yourself (by reversing polarity). Alas, there's a bit of trial and error involved when connecting RS485 lines; usually, A is negative and B is positive, but on some devices, it could be the opposite.
Now, to the next post -- how to break data into packets in the log. For simple visual aids, I'd recommend starting with the Regex Markup feature. It's very simple and could work quite well in your case, actually. Each packet is prefixed with a fixed preamble 0x55 0xff, so you can define a regex as \x55\xff and set "Markup Mode" to "Add delimiters before matches". All packets will be visually separated from one another with red lines.
0x55 0xff
Now, if we are not talking about visual aids but rather want to export it into Wireshark as a sequence as separate packets, it's a different story. Here, you would need to write a custom script (using any language of your choise) that would parse the data from .njlog file (generated by IO Ninja), split it into packets, then write those to a .pcap file programmatically.
.pcap
IO Ninja only knows how to generate a .pcap file when it captures actual network packets via Pcap Sniffer, Ethernet Tap and similar plugins. For Serial Tap and most other plugins, however, the data is a continuous data stream, and it doesn't map to the .pcap format very well -- at least, not well enough to do it automatically. However, with a custom script everything is possible. If you choose this path -- the .njlog file format is very simple, and all the related structure definitions are open source:
<ioninja-dir>/scripts/api/log_RecordCode.jnc <ioninja-dir>/scripts/api/log_RecordFile.jnc
<ioninja-dir>/scripts/api/log_RecordCode.jnc
<ioninja-dir>/scripts/api/log_RecordFile.jnc
Finally, your last question is about how to use Jancy to work with BACnet MSTP. It's also possible, of course, but let's clarify your end goal here. Do you want to (a) conveniently prepare packets for sending -- i.e., fill the fields using the property grid, automatically calculate checksums, etc? Or (b) parse the incoming data and show the decoded packets in the log?
For (a) you would want to create a packet template. Here's a simple introduction and tutorial: https://ioninja.com/doc/developer-manual/tutorial-ias-packet.html. Also, for your reference, you can take a look at the Modbus Packet Template library (Packet Template Pane -> Load Stock Script -> Modbus RTU, sources are at /scripts/packets/ModbusRtu.jnc).
Modbus RTU
/scripts/packets/ModbusRtu.jnc
If it's (b) that you want, then you need a protocol analyzer plugin. This is a bit more challenging task. Here's a short tutorial: https://ioninja.com/doc/developer-manual/tutorial-plugin-analyzer.html to get you started. Also, I'd recommend skimming through the sources of the Modbus Analyzer plugin for a working real-world protocol analyzer. Sources could be found at /scripts/plugins/Modbus/
/scripts/plugins/Modbus/
Let me know which one (a or b) is your priority, and we'll focus on that. Maybe, we can create a quick skeleton script for you to work on.
Thanks for your feedback!
The high DPI support in IO Ninja depends on QT -- it's mainly about the version of QT IO Ninja is linked to. Before ioninja-5.4.0, we used qt-5.6 on Windows. qt-5.6 has very poor support for mixed-DPI setups. I know it might look OK on your monitor -- but try plugging your 4k into a laptop and setting 100% on laptop, 150% on the monitor. Drag IO Ninja from one monitor to the other, and you'll see that the visuals basically fall apart; there were glitches, overlapping, broken layouts, etc.
ioninja-5.4.0
qt-5.6
Now, we link to qt-5.15.10, which objectively has much (much!) better support for high DPI -- and mixed DPI configurations in particular. It's still not perfect yet, I must admit, but it's a huge step forward in this department. Also, QT gradually improving; they still keep releasing updates to QT5 (eventually, we will port things to QT6, of course) -- so it's reasonable to expect high DPI support to get better, as well.
qt-5.15.10
The effect you mentioned (icons & toolbar looking too big) is due to the fact that the 150% scaling has its own quirks. It's next to impossible to automatically scale non-vector images nicely by the x1.5 factor. And QT doesn't even try to. If you change the Windows display scaling, you can notice that icons in 150% and 200% scaling are the same (scaled x2), while fonts are scaled differently. This causes certain distortions (and this "something's off" feeling about icons & toolbar). We can experiment with turning scaling off just for icons, but the ultimate solution is either (a) move to vector icons (probably not) or (b) provide multiple sets of icons for different scaling factors (x1, x1.5, x2). I can't promise to have these improvements in the very next release, but soon -- high DPI is getting more and more widespread.
Hope this clarifies the issue and makes sense.
The scripts folder is a part of every installation package (i.e., <ioninja-root-dir>/scripts/).
scripts
<ioninja-root-dir>/scripts/
The scripting language is Jancy, but it's a form of a safe scripting dialect of C/C++, so you should have no problems reading it.
Hello Dan,
If you want to filter data flowing in one direction (i.e., leave TX-only or RX-only), you can apply the TX/RX Filter.
Beware, that with the Pipe Monitor plugin, each block of data will be visible twice (i.e., once as TX for one side of the connection, and once as RX for the other side). If you want to get rid of this duplication, you can first filter the log using the file ID (i.e., first, leave only data visible by one side of the connection) -- and then apply the TX/RX filter to leave TX-only or RX-only.
And of course, you can post-process the captured .njlog log file in any custom way you want -- with a script written in any language of your choice. The log file format is very simple and all relevant definitions are open-source:
scripts/api/log_RecordFile.jnc scripts/api/log_RecordCode.jnc
scripts/api/log_RecordFile.jnc
scripts/api/log_RecordCode.jnc
Thanks for the information. We probably should add this driver to the exception list and automatically select the more compatible (but less performant) read mode Check COMSTAT.cbInQue for it.
Check COMSTAT.cbInQue
So, just to double-confirm -- after selecting this mode (or reducing Read Parallelism to 1), you can use the Serial plugin in IO Ninja to work with this device, right?
1
From the log we can see the port getting opened (so it's not a permission issue, and the Administrative access is not needed). Then immediately a "The requested resource is in use" error (winerror code ERROR_BUSY) follows.
"The requested resource is in use"
ERROR_BUSY
This most likely means that concurrent read operations are not supported by the driver. By default, IO Ninja is issuing multiple simultaneous read operations on most IO transports to maximize throughput. However, some serial drivers can't handle multiple reads at the same time.
Luckily, IO Ninja has means to overcome this limitation of such drivers. To combat that, you can either (a) decrease Read parallelism to 1 or (b) set Read mode to Check COMSTAT.cbInQue
Just curious, what kind of serial port is that (vendor/model)? You can hover your mouse over the item in the port dropdown box, and a tooltip with in-depth details on the port will pop up.
Hi.
Just tried it on linux-6.6.1-arch-x86_64, and it works (showing both TX and RX data). What exactly are the symptoms? Can you share any screenshots? Also,
linux-6.6.1-arch-x86_64
$ uname -a ?? $ ./tdevmon --version ??
Hello Daniel,
(1) Even if the IO ninja is not powered, pass-through will be enabled, so data will still pass
The Ethernet Tap is actively forwarding packets from one port to the other; without power, the connection will break. Not easy to fix; we plan to address that in the next iteration of the HW development cycle, but at the moment, it is what it is.
(2) Work with shielded cables
We don't specifically handle shielded cables; most likely, will work, but needs testing in your specific setup.
(3) Automatic mode, so when it starts up, it must be capturing data automatically (unattended)
This is all up to how you set up your Linux box.
(4) Possibility to create a circular log on the linux device, with specific size in Mb, ex 50 Gb or similar.
Not currently, but the rotation of size-limited logs can be added to ioninja-hwc (the command-line tool).
ioninja-hwc
(5) Possibility to run everything completely headless on the linux device, preferably as a linux service
ioninja-hwc runs on any Windows/macOS/Linux i386/x86_64/ARM/AArch64
(6) Possibility to use filters in this headless mode
Support for pcap-filter can easily be added to ioninja-hwc.
pcap-filter
If you can't or don't want to wait for when we add (4) or (6), no problem -- our USB protocol is open-source so that you can implement your own reader from Ethernet Tap in any programming language you choose.
So if (1) is not a deal-breaker, then it's possible.
Not exactly a bug but rather a missing misuse protection.
So, what's happening? The automatic calculation of Modbus TCP frames is done like this:
void updateModbusTcpLength(void* p) { size_t size = dynamic sizeof(p); if (size < sizeof(io.ModbusTcpAduHdr)) return; ((io.ModbusTcpAduHdr*)p).m_length = size - offsetof(io.ModbusTcpAduHdr.m_deviceAddress); }
Basically, m_length is set to the length of the payload (from m_deviceAddress and to the very end of the packet).
m_length
m_deviceAddress
In your case, the packet length is 17, offset of m_deviceAddress is 6, so m_length is set to 11. However, the size of Modbus TCP read should always be 12 (not 17). If you erase the trailing 5 bytes from your packet, it will work as expected.
Modbus TCP read
That should solve the issue; for more relevant details, read on.
By design, packet templates never allow creating a packet shorter than the template itself. However, it's allowed to add any suffix after the templated headers (that would be required for Modbus TCP write, for instance).
Modbus TCP write
In ioninja-5.1.0 we added the fixedSize attribute exactly to prevent unintentional misuse in case of packets that should never grow longer than the template itself.
ioninja-5.1.0
fixedSize
Modbus TCP read is exactly this kind of packet, but in the current version of IO Ninja, it's missing the fixedSize attribute. We will add it in the next release of IO Ninja, but in the meantime, you can fix it by yourself. To do so, please open ioninja/scripts/packets/ModbusTcp.jnc in any text editor, locate the definition of ModbuTcpReadPacket (in the very beginning), and modify it as such:
ioninja/scripts/packets/ModbusTcp.jnc
ModbuTcpReadPacket
[ displayName = "Modbus TCP read", fixedSize ] struct ModbusTcpReadPacket { // ...
In order to apply it, please reload this template into the Packet Template editor (Edit packet template -> Load stock script -> Modbus TCP).
Edit packet template
Load stock script
Modbus TCP
Let me know if this works for you.