Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.
Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).
@jose-marro
I was thinking of using it for parsing and generating frames (IEC60870-5-103 frames)...
For parsing frames, dylayout would be a perfect tool.
dylayout
For generating frames programmatically, dynamic layouts won't add much extra convenience (and dyfields are currently all const anyway).
dyfield
const
But! One thing I didn't mention above is that dynamic layouts are also perfect for creating packet templates!
After defining a dynamic layout "specification" for a protocol, you will be able to conveniently build packets for this protocol in the Property Grid on the Hex Transmit pane in IO Ninja. Set enumeration fields via drop-down lists, set bits in bitfields with check-boxes, have big-endian automatically converted for you, etc. If a protocol uses checksums, you can define methods for automatic calculation of those checksums before transmission.
All in all, it's an awesome tool for generating and sending out test packets! If you didn't see it, please check it out -- it could be just what you are looking for.
A short follow-up after thinking more about q1.
In my previous write-up, I used the word "packets" when talking about binary blobs that dynamic layouts work with. But of course, those could be any binary objects — disk files, disks themselves, shared memory, etc.
Indeed, with packets, we usually generate the whole thing from scratch—and using dynamic layouts here doesn't offer much.
But if we think about objects like files or disks, it makes perfect sense to allow modification alongside parsing. Something like (1) locate a specific field inside a file, (2) modify this field, (3) proceed to the next one.
So yeah, I think we should remove the forced const on dyfield declarations. We still need to somehow preserve const-correctness for the parse-only cases, though.
One way would be to introduce an auxiliary class jnc.MutableDynamicLayout, which would take non-const pointers; when the dylayout argument is jnc.MutableDynamicLayout, the Jancy compiler won't add implicit const to dyfield declarations.
jnc.MutableDynamicLayout
Thoughts?
P.S. Moved the topic to General Discussion
Hello Josep,
Happy to see you are trying to play with the new Jancy feature! Indeed, dynamic layouts are replacing dynamic structs -- which never were utilized in IO Ninja (as they were way too limited for practical use).
With this new approach, it's possible to describe pretty much any protocol or protocol stack. Please check the release announcement; there, I outlined the main problems that dynamic layouts really help with, which boils down to this:
To answer your questions:
q1. is it possible to assign a value to a "dyfield"?
TLDR: currently, no. In theory, yes, but that (probably) would be a misuse.
The main motivation for dynamic layouts was a simplification of binary packet parsing. Therefore, jnc.DynamicLayout expects a read-only void const* as a buffer pointer, and the Jancy compiler adds an implicit const to all fields to reinforce that.
jnc.DynamicLayout
void const*
In theory, it's possible to remove this limitation and allow passing non-const buffers to the dylayout statement (thus allowing modification of dyfield items). That shouldn't really break anything, but dynamic layouts won't really provide many benefits for the generation of packets (as opposed to parsing). The difference is that when we generate a packet, we outright know what has to be in the packet. So why not take a std.Buffer and append all the necessary blocks one by one?
std.Buffer
append
q2. is it possible to access to the "DynamicLayout" elements outside of the dylayout section?
TLDR: yes and no (can enumerate all the fields, but can't reference a particular one by name or index).
Things like layout.myChar are not possible, even in theory. What if there's the branch where myChar is defined was simply skipped? Worse yet, what if myChar's type depends on the branch, like:
layout.myChar
myChar
dylayout (layout) { dyfield uint8_t bitness; switch (bitness) { case 8: dyfield uint8_t myChar; break; case 16: dyfield uint16_t myChar; break; case 32: dyfield uint32_t myChar; break; case 64: dyfield uint64_t myChar; break; } }
On the other hand, you can iterate over all the discovered fields after exiting from dylayout -- that's what IO Ninja does to represent packets in the log. To do so, you pass jnc.DynamicLayoutMode.Save to the jnc.DynamicLayout constructor and then recursively walk over sections of jnc.DynamicLayout.
jnc.DynamicLayoutMode.Save
Here's a rather lengthy but realistic example. To run it, simply glue all 3 code snippets below together.
First, let's define the protocol structures:
pragma(Alignment, 1); struct EthernetHdr { uint8_t m_dstMac[6]; uint8_t m_srcMac[6]; bigendian uint16_t m_etherType; } struct IpHdr { uint8_t m_headerLength : 4; uint8_t m_version : 4; uint8_t m_typeOfService; bigendian uint16_t m_totalLength; bigendian uint16_t m_identification; bigendian uint16_t m_flags : 3; bigendian uint16_t m_fragmentOffset : 13; uint8_t m_timeToLive; uint8_t m_protocol; bigendian uint16_t m_headerChecksum; bigendian uint32_t m_srcAddress; bigendian uint32_t m_dstAddress; } struct TcpHdr { bigendian uint16_t m_srcPort; bigendian uint16_t m_dstPort; bigendian uint32_t m_seqNumber; bigendian uint32_t m_ackNumber; uint8_t m_reserved : 4; uint8_t m_dataOffset : 4; uint8_t m_flags; bigendian uint16_t m_window; bigendian uint16_t m_checksum; bigendian uint16_t m_urgentData; }
Now, here comes the main function. The dylayout part is the heart of the parser. If you want pause-and-resume, you should put it into an async coroutine -- then it will be possible to pause in the middle of parsing the packet if it's not complete yet -- and wait for more bytes. But for the TCP/IP stack, it won't make much sense, of course.
async
int main() { // a sample packet char packet[] = 0x"00 1d aa 5f 9c 68 00 ad 24 90 be ae 08 00 45 00" 0x"00 34 63 aa 40 00 80 06 00 00 c0 a8 01 79 14 bd" 0x"ad 18 83 53 01 bb 77 02 38 0b 00 00 00 00 80 02" 0x"fa f0 bc 0c 00 00 02 04 05 b4 01 03 03 08 01 01" 0x"04 02"; jnc.DynamicLayout layout( jnc.DynamicLayoutMode.Save, // when parsing, also save all discovered fields packet, sizeof(packet) ); dylayout (layout) { // the main specification dyfield EthernetHdr hdr; switch (hdr.m_etherType) { case 0x0800: // IP4 dyfield IpHdr ipHdr; ipHdr.m_protocol = 6; if (ipHdr.m_headerLength * 4 > sizeof(IpHdr)) // have options dyfield uint8_t options[sizeof(IpHdr) - ipHdr.m_headerLength * 4]; switch (ipHdr.m_protocol) { case 6: // TCP dyfield TcpHdr tcpHdr; break; case 17: // UDP case 1: // ICMP // etc } break; case 0x86dd: // IPv6 case 0x0806: // ARP // etc } } printGroup(packet, layout); return 0; }
Finally, here's how to do a recursive walk across all discovered items. A more sophisticated version of such walker could be found in scripts/common/log_RepresentDynamicLayout.jnc (it's used to render dynamic layouts in the log with respect for color, format specifier, display name, and other attributes):
scripts/common/log_RepresentDynamicLayout.jnc
string_t g_indentStep = " "; void printGroup( void const* p, jnc.DynamicSectionGroup* group, string_t indent = "" ) { for (size_t i = 0; i < group.m_sectionCount; i++) { jnc.DynamicSection* section = group.m_sectionArray[i]; switch (section.m_sectionKind) { case jnc.DynamicSectionKind.Array: printf("%08x%s %s %s[%d]\n", section.m_offset, indent, section.m_type.m_typeString, section.m_decl.m_name, section.m_elementCount); break; case jnc.DynamicSectionKind.Struct: jnc.StructType* type = dynamic (jnc.StructType*)section.m_type; printFields(p, section.m_offset, type, indent); break; case jnc.DynamicSectionKind.Group: printf("%08x%s %s {\n", section.m_offset, indent, section.m_decl.m_name); printGroup(p, section, indent + g_indentStep); printf("%s}\n", indent); break; } } } void printFields( void const* p, size_t baseOffset, // struct field offsets are relative to the beginning of the struct, so we need base offset jnc.StructType* type, string_t indent ) { for (size_t i = 0; i < type.m_fieldCount; i++) { jnc.Field* field = type.m_fieldArray[i]; size_t offset = baseOffset + field.m_offset; printf("%08x%s %s %s", offset, indent, field.m_type.m_typeString, field.m_name); if (field.m_type.m_typeKind != jnc.TypeKind.Struct) printf(" = %s\n", field.getValueString(p + offset)); else { printf("\n"); printFields(p, offset, dynamic (jnc.StructType*)field.m_type, indent + g_indentStep); } } }
But what if you want to access a particular field instead of walking across all fields? Then you need to access it within dylayout, from the branch where this field is visible! Otherwise, the field you try to access may be missing or be of the wrong type.
A short summary.
I know, it's a lengthy reply, but hope this makes sense. Feel free to follow up with any questions or suggestions!
@schunsky
Is it possible to get the "merged" data when I write a custom protocol analyzer?
This merging strategy is a part of the logging engine, so yes, it applies to all kinds of plugins, including custom protocol analyzers.
I concluded that the strange behaviour where I received data 1 byte each was because of an USB-UART adapter I used.
Hmm, a particular model of USB-to-UART is unlikely to cause this one-byte-at-a-time behavior. My guess is that the buffering settings are to blame (i.e., IO Ninja reads into a one-byte buffer). Check the "Buffering & compatibility" section in properties and try resetting it all to defaults.
Adding a shortcut for clear-the-log is technically trivial. But it should be something that's really hard to press by accident (something like Ctrl+Shift+F8). I mean, imagine a user confusing the shortcut and accidentally killing a log that was built overnight! We even had an opposite (kind of) to your feature request—having a confirmation dialog for clear-the-log!
clear-the-log
Ctrl+Shift+F8
Adding shortcuts to start-capture, stop-capture, connect, disconnect, etc. -- is technically harder because all those commands are plugin-specific and created from the plugin scripts. Hence, plugins should be able to assign shortcuts to the actions they create -- but IO Ninja currently doesn't have such an API (which should be added, of course).
start-capture
stop-capture
connect
disconnect
Overall, a totally valid feature request! We'll try to get something in this department for the next release...
Shun,
By default, IO Ninja merges all blocks of the same data stream together (TX to TX, RX to RX) and highlights merged block boundaries using this grey-white checker pattern. You can turn off this highlighting and configure other details of the merging strategy here:
From your screenshot, I can see that (1) data arrives one byte at a time and (2) not all RX blocks are merged together.
(2) means that you modified the merging strategy (e.g. set to a 20ms threshold or something like that)
(1) most likely, it's caused by the custom buffering rules (e.g. the read block size is set to 1 byte). Unless you have a specific reason to do otherwise, it's recommended to use default buffer sizes.
Hi again, Jörg,
Correct, for capturing, elevation is required. This is by design -- capturing USB traffic is a major security threat (e.g., it allows intercepting keystrokes, mouse movements, etc). The USBPcap driver makes sure only elevated processes can open \Device\USBPcap<n> for reading by assigning a corresponding security descriptor. And there's no way to override this behavior -- the security descriptor is hardcoded.
\Device\USBPcap<n>
Hi Jorg,
If the USB Mon device list is empty, it most likely means that USBPcap is not properly installed (did you reboot after installing it?) The enumeration of USBPcap devices requires no elevated privileges, so you should see the list of available USB devices (together with their mapping to particular USBPcap devices) in the drop-down list no matter if you run IO Ninja elevated or not. You can try running the USB Mon session as Administrator for the sake of the experiment, but it shouldn't change much...
Can you see the list of available USBPcap devices in Wireshark by the way?
Glad to hear it helped! Let me know if you have more questions regarding this protocol analyzer of yours. Which protocol is that, by the way?
There're new tutorials on writing custom protocol analyzers and packet templates using dynamic layouts; please check it out:
https://ioninja.com/doc/developer-manual/tutorial-plugin-analyzer.html https://ioninja.com/doc/developer-manual/tutorial-ias-packet-dylayout.html
(The original tutorial is still available at https://ioninja.com/doc/developer-manual/tutorial-plugin-analyzer-legacy.html)
It's now much easier to write customer protocol analyzers than it's used to be...
@bartho-dröge
Uhm, not quite sure what you mean? It's actually OK to download IO Ninja packages even without signing in to ioninja.com
Serial Terminal requires the capability org.jancy.io.serial, yes. But still, it doesn't stop users from downloading and running evaluation.
org.jancy.io.serial
But wait, in your case, you have an active subscription, so everything should be unlocked and available. Please let me know if it's not the case and you have problems accessing some functionality...
Hi Shun,
I'd say it's a better practice to keep state variables as member fields of your parser class rather using globals. But it's not written in stone, of course.
To reset state vairables/fields on every session start, process log.StdRecordCode.SessionStart in your log.Converter.convert(...) routine and do initialization from there. Also, you want to override log.Converter.reset() and do initialization here, too (reset() will get called when a user force-clears the log, rebuilds the log by adding a filter, etc).
log.StdRecordCode.SessionStart
log.Converter.convert(...)
log.Converter.reset()
reset()
You can check the official Modbus Analyzer for reference (there are two versions now, a new one based on dynamic layouts and the legacy once).
Also, there're new tutorials on writing protocol analyzers and packet templates using dynamic layouts, check it out:
IO Ninja can work with any USB-to-Serial adapter via the Serial Terminal plugin.
As for other hardware Serial sniffers, we have plugins for EZ-Tap Pro/VersaTap and for any generic dual-COM tap.
You can run a free evaluation to try and see if everything works for you before purchasing; let me know if your evaluation is over and you need to extend it.
Glad to hear the binary data settings worked for you! Although I can't see the screenshot this time...
Regarding your question -- the Serial Terminal plugin requires org.jancy.io.serial.
Actually, all IO Ninja feature pages on our website show the capability requirements at the very top (as well as the status of a particular feature on your account):
Also, if a particular feature is locked, and you try to use it from the app -- a pop-up with the required capability should appear (but you have an active evaluation now, so all capabilities are unlocked).
Re 7bits vs 8bits -- understood. Alright, so 2400/7/N/2 it is!
Is there a way to increase the number of hex bytes and translation to ASCII in order to show more data on one line?
Absolutely. IO Ninja is highly configurable in what concerns binary data display. You can increase the number of bytes per hex-view line -- or maybe try the Plain-text view. This could actually work even better in your case because you don't have any unprintable characters except for CR/LF.
All this can be configured on the Settings screen:
Switching between hex-view and plain-text can also be done from the main app window:
Hi Tom,
From the second screenshot, it's quite apparent that the incoming and outbound data are byte-to-byte the same. Most likely, you incorrectly wired it so that the same data is being fed to both the TX and RX connectors of the Serial Tap. Could you please share the wiring diagram (or a photo of how you connect your Tap)?
Also, are you sure about the correctness of your serial settings? The { 7 data bits + 2 stop bits } combo is much less common than { 8 data bits and 1 stop bits }.
Hi Massimo!
Indeed, the filter is currently implemented in the session only -- it should be transferred to the log plugin instead; this way, it will be there when you open a standalone log. We definitely will fix this in the upcoming release (scheduled for May). In the meantime, here's a workaround:
Create a fresh Ethernet Tap session and save it (let's say, in $HOME/ethernet-tap-1). This will create 3 files under $HOME/ethernet-tap-1:
$HOME/ethernet-tap-1
ethernet-tap-1.njssn
ethernet-tap-1.njlog
ethernet-tap-1.njcfg
Close the session. Then, using any file manager, copy your original log file over ethernet-tap-1.njlog. Re-open this session, and it will use the original log file. Now filtering packets will be available -- together with all other actions of the Ethernet Tap session (e.g., capturing more data and appending it to the log).
Hope this works for you.
Once again, the issue with not being able to filter standalone Ethernet Tap/Pcap Sniffer logs will fixed in the next release.
I think it would be a very helpful feature to add to IO ninja to just be able to save a log as a raw binary file for each direction.
It's very easy to achieve with a log filter/observer plugin script.
However, I can't say I find it very useful to have two raw binary files containing just the raw bytes flowing in each directing -- how would you reconstruct the mapping between requests and replies in case of some non-trivial conversation between the client and server?
I often have serial port logs that I just want to save to a bin file as-is, and I end up closing IO ninja and just going back to something like CoolTerm which supports this natively.
If it's a uni-directional log kind of thing (e.g., your device keeps dumping debug printfs to a RS232, and you want to save it to a file) -- this can be done using Session Linking. Open a Serial session, open a File Stream session in Write-only mode and point it to some file, then "link" those sessions. Everything that the Serial session reads from the device, will be redirected to your file.
it also supports recording directly to a file, in case the computer crashes or powers off during logging
Actually, IO Ninja always keeps its logs in disk files. When you "Save Log", the temporary .njlog file simply gets copied into the file you select. But if you "Save Session", those .njlog files will get stored in the specified folder, and you will know exactly where to find them in case of a crash.
.njlog
But once again, a dedicated plugin that would explicitly save RX to one file and TX to another is quite easy to implement. I don't know, maybe we should add it to one of the upcoming releases as an official plugin (or a sample)?
Try the updated analyzer:
BacNetMsTp.7z
It checks CRCs now and discards frames with broken headers (in the previous log, the decoder didn't actually stop, but because a broken header specified a very long payload size, it kept buffering data assuming it still was a payload).
Also, when you upload the log, please detach the Analyzer before saving the log. This way, we can access the original raw & unprocessed data (instead of the already decoded frames).
No prob, I've increased the upload file size limit to 8MB.
Also, you could have archived it with 7z
7z