Packet template 'alignment' issue

Thank you very much, it works now with the updated syntax. Would be great if the tutorial would be updated too :-)...
In the protocol we are trying to trace with the IO Ninja, each packet has a different size. According to the packet type/size the data structure is different. Is it possible to create such a template that can display different data structure according to the packet type ?
If so, can you send me an example of such a template.
If not, is there a way to do ?

Here is how out packet looks like:
pragma(Alignment, 1)

struct FE_BE_Packet
{
uint8_t StartOfPacket;
uint16_t Length;
uint8_t MessageID;
uint16_t GlobalSequenceNumber;
uint16_t MessageSequenceNumber;
char MessageData[70];
uint16_t CheckSumData;
}
The 'MessageData' is the one that keeps changes (size and content)

Is it possible to create such a template that can display different data structure according to the packet type

Just to ensure we're on the same page -- packet templates are for editing packets before transmission, not for displaying them in the log. After defining a packet template, you can conveniently prepare it with a property grid before transmission.

If that's what you try to do, then the answer is YES, you can define a packet template with a variable-sized payload length and a checksum in the end:

import "crc16.jnc"

pragma(Alignment, 1)

struct FE_BE_Packet {
	uint8_t StartOfPacket;
	uint16_t Length;
	uint8_t MessageID;
	uint16_t GlobalSequenceNumber;
	uint16_t MessageSequenceNumber;

	[ 	
		userAction = "Update checksum",
		autorun = "Auto-update checksum" 
	]
	void updateChecksum() {
		size_t size = dynamic sizeof(this) - sizeof(uint16_t); // without checksum
		uint16_t* checksum = (uint16_t*)((char*)this + size);
		*checksum = crc16_ansi(this, size, -1); // Modbus CRC
	}
}

Here is a quick rundown of what's happening here.

The structure definition itself is for the fixed-size header -- the header fields can be edited with a property grid. Beyond this header, you can enter a variable-sized payload in the hex editor.

Now, to the interesting part. We want to auto-update the checksum every time the user changes any fields in the header (or the payload itself). To achieve that, I create a method called updateChecksum(), mark it as userAction (so it's user-accessible via a hyperlink) and autorun (execute on every data change) using function attributes [...], and inside this method, I calculate a checksum and write it to the last two bytes of the packet. In my example, I use the Modbus checksum (CRC16-ANSI with the seed of 0xffff), but feel free to update it according to your needs.

Hope this helps!

Thanks for your reply.
From what you wrote, i understand i'm using the wrong tool, since i want to analyze incoming packets and log then to a log or display them at runtime, and not build packets before sending them...
Can you please suggest a way to do so, and provide an example i can use ?
Thanks

If you want to parse the captured data streams and inject human-readable messages alongside the original data, you need a protocol analyzer. This is, however, a more challenging task than writing a packet template.

IO Ninja features open-source Modbus and DF1 analyzers that could be used as a reference for approaching this task. Also, there's a tutorial in the Developer's Manual:

https://ioninja.com/doc/developer-manual/tutorial-plugin-analyzer.html

Thanks for your reply.
I'm looking at the tutorial you pointed to, on how to create a protocol analyzer.
However, i could find in the IO-Ninja IDE how to create a new IONinja project... there is no menu 'File->New Project' only 'File->New Session...'. Was the IDE changed recently ? Can you please guide me how to do it with the current IONinja IDE.
Thanks

Hi,
I managed to solve the issue.
There is a typo in the tutorial page (https://ioninja.com/doc/developer-manual/tutorial-plugin-analyzer.html).
The user should open NetBeans IDE for creating a new project, and not the IO Ninja IDE...
Please update.

Actually, there's no typo; IO Ninja IDE is a set of NetBeans plugins, as explained on the Download page:

Ok, to me it was very unclear, and took me a while to figure that out...

In any case, i was able to create a new protocol analyzer according to the help provided in - https://ioninja.com/doc/developer-manual/tutorial-plugin-analyzer.html
I've added the plugin to IO Ninja (using the settings dialog), and add the new plugin as a layer on top of 'Serial Tap'.
We are using a IO Ninja serial Tap device to sniffer serial packets (we use TTL).
I'm able to see the packet flow when the serial tap is connected, however, i wasn't able to see the protocol analyzer plugin working... the plugin parses the incoming data, and when a full packet is completed is sends a formatted string to the logger (just as done in the tutorial).
Is there something i'm missing ?
Many thanks.

Ok, to me it was very unclear, and took me a while to figure that out...

Installing and configuring NetBeans is an extra step, true. But I thought the explanation on the Download page or the dedicated article in the Developer Manual address this issue... If not, how would you rephrase those explanations to make things more clear?

...however, i wasn't able to see the protocol analyzer plugin working

Most likely, something is wrong with the implementation of your protocol analyzer. You didn't describe which protocol you try to parse, how exactly you do that, and which data you feed to the analyzer.

If you upload the sources of your protocol analyzer somewhere, we can take a look.

BTW, the tutorial has a link to an archive with a working analyzer of a made-up protocol. There are also real-life Modbus Analyzer and DF1 Analyzer; all those can be used as a reference when developing one of your own.

Hi,
Thanks for your reply, indeed i had a parsing bug in my code... fixed it and now i'm able to see parsed packets in the log .
I'm using the log.representStruct to print a stucture in the log, cool stuff !!!
Say you have a structure that contains another stucture and so on for a few levels. Is there a way to use the 'representStruct' recursively, so all inner struct will be formatted too ?

Hello Chaim,

Glad to hear you've made some good progress with your protocol analyzer!

Yes, log.representStruct should simplify and at the same time "standardize" representation of binary-based packets. And yes, you are right -- at the moment, it doesn't recursively represent all the nested structs. That's because this functionality was never required in the stock protocol analyzers in IO Ninja -- we use "layered" representation rather than "nested" because (a) this makes resulting tables more compact and easier to navigate, and (b) allows us to colorize layers using different colors. But there's nothing that prevents adding the support for nesting to log.representStruct; I think we can add that to the upcoming release.

How deep is the nesting in your case? Could you please provide definitions of the packet structs in your protocol?

Just FYI, the implementation of log.representStruct is in scripts/common/log_RepresentStruct.jnc; you can try experimenting with it by yourself until the official release.

Hi,
I tried to modify the 'scripts/common/log_RepresentStruct.jnc' to support nested structures with no success.
Can you please send me a modified version of 'scripts/common/log_RepresentStruct.jnc' that supports nested structures.
It will be a big help.
Thanks

No prob: log_RepresentStruct.jnc

Just FYI, here's the added part (special handling for struct fields):

/// ...
		if (field.m_type.m_typeKind == jnc.TypeKind.Struct) {
			representation.addHyperText($"%1%2:\t"(indent.m_sz, displayName));
			representStruct(
				representation,
				dynamic (jnc.StructType*)field.m_type,
				displayName,,
				p + field.m_offset,
				baseOffset + field.m_offset,
				0, 0, // no [+/-] header, always expanded
				indentLevel + 1
			);

			continue;
		}
/// ...

Having sub-fields collapsible is theoretically possible but won't scale well -- we have 32 total fold-flags per record, so it's better to assign those statically (rather than from a loop in a recursive function). But I think having struct fields always expanded should be good enough in most cases.

Hi Again,
Thanks for the above scripts it really works great !!!
I've presented the protocol analyzer i did to a big forum in out company, and a few question came up.

1. We would like to add this tool as part of our nightly build and validation process. Once a new build is created and deployed to a target, we'd like a way to run the IONinja via a batch file (running some command lines), load the protocol analyzer open the port and start recording the session. Is it possible to run the IONinja tool via an external API /commandline ? if so, can you send me an example how ?
2. The protocol analyzer is written in Jancy language. Is there an ability to access in Jancy APIs from DLLs created in C++ or Python ?
3. Can you activate in Jancy a batch file and do some manipulation on the bytes received ? i.e. pass the received bytes as an argument to a batch file, and get an answer from the batch file ?
   Many thanks
   Chaim

Hi again, Chaim,

From the first screenshot, I assume, you are running on Windows, correct? Then I will use Windows-style paths and declarations in the samples below.

1. Command line

Yes. You can open the necessary plugin(s) and start capturing from the command line. There are multiple ways of achieving that, and the easiest would be this.

First, configure the session the way you need -- i.e., start the main plugin (e.g., Serial Tap), attach required analyzer plugins (e.g., Modbus RTU, or your own analyzer), and configure all the necessary settings using the UI. Then, save the session somewhere (e.g., C:/Projects/playground/ioninja/my-analyzer-session/my-analyzer-session.njssn)

Now, you can start IO Ninja passing it a path to .njssn to automatically open this session. To automatically start capturing, also pass -c (i.e., connect):

ioninja -c <path-to-njssn>

2. FFI (Foreign Function Interface)

Yes. One of Jancy design goals was easy and efficient interoperability with C/C++ code. Again, multiple ways of calling external code exist, the most straightforward would be declaring an interface to your C/C++ library via the dynamiclib construct.

Let's say, you have a library called my_lib.dll with the following exported C function:

extern "C"
__declspec(dllexport) 
size_t 
my_inverse(
	void* out,
	size_t outSize,
	const void* in,
	size_t inSize
) {
	// inverse the in-buffer and write it to out-buffer
}

In your Jancy code, declare an interface to this library like this:

pragma(ThinPointers, true);

dynamiclib MyLib {
	size_t 
	my_inverse(
		void* out,
		size_t outSize,
		const void* in,
		size_t inSize
	);
}

pragma(ThinPointers, default);

Basically, function declarations are the same as in C/C++; you only have to make sure to use "thin" pointers.

Jancy pointers are "fat" -- they include extra information about region, type, etc. To achieve ABI compatibility with C/C++, we need to explicitly declare pointers as "thin", like this:

char thin* p;

If there could be many FFI declarations -- like in an interface to a C/C++ library -- it may be more convenient to implicitly treat ALL pointers as "thin" using pragma(ThinPointers, true).

Now, when you need to call your library, load and call it like this:

MyLib myLib;
myLib.open("path-to-your-lib/my_lib.dll");
myLib.lib.my_inverse(p1, size1, p2, size2);

3. Creating an external process is possible, too, of course. However, passing binary data to a process via the command line -- and especially getting an answer back -- would be much less convenient and efficient. If you need to perform some data processing in external code, I strongly recommend using the method shown above, i.e., create a C/C++ library and call it from Jancy. It's both easy to implement and as efficient as it gets in runtime.

Hope this helps!