Packet template 'alignment' issue

Thanks for your reply.
From what you wrote, i understand i'm using the wrong tool, since i want to analyze incoming packets and log then to a log or display them at runtime, and not build packets before sending them...
Can you please suggest a way to do so, and provide an example i can use ?
Thanks

If you want to parse the captured data streams and inject human-readable messages alongside the original data, you need a protocol analyzer. This is, however, a more challenging task than writing a packet template.

IO Ninja features open-source Modbus and DF1 analyzers that could be used as a reference for approaching this task. Also, there's a tutorial in the Developer's Manual:

https://ioninja.com/doc/developer-manual/tutorial-plugin-analyzer.html

Thanks for your reply.
I'm looking at the tutorial you pointed to, on how to create a protocol analyzer.
However, i could find in the IO-Ninja IDE how to create a new IONinja project... there is no menu 'File->New Project' only 'File->New Session...'. Was the IDE changed recently ? Can you please guide me how to do it with the current IONinja IDE.
Thanks

Hi,
I managed to solve the issue.
There is a typo in the tutorial page (https://ioninja.com/doc/developer-manual/tutorial-plugin-analyzer.html).
The user should open NetBeans IDE for creating a new project, and not the IO Ninja IDE...
Please update.

Actually, there's no typo; IO Ninja IDE is a set of NetBeans plugins, as explained on the Download page:

Ok, to me it was very unclear, and took me a while to figure that out...

In any case, i was able to create a new protocol analyzer according to the help provided in - https://ioninja.com/doc/developer-manual/tutorial-plugin-analyzer.html
I've added the plugin to IO Ninja (using the settings dialog), and add the new plugin as a layer on top of 'Serial Tap'.
We are using a IO Ninja serial Tap device to sniffer serial packets (we use TTL).
I'm able to see the packet flow when the serial tap is connected, however, i wasn't able to see the protocol analyzer plugin working... the plugin parses the incoming data, and when a full packet is completed is sends a formatted string to the logger (just as done in the tutorial).
Is there something i'm missing ?
Many thanks.

Ok, to me it was very unclear, and took me a while to figure that out...

Installing and configuring NetBeans is an extra step, true. But I thought the explanation on the Download page or the dedicated article in the Developer Manual address this issue... If not, how would you rephrase those explanations to make things more clear?

...however, i wasn't able to see the protocol analyzer plugin working

Most likely, something is wrong with the implementation of your protocol analyzer. You didn't describe which protocol you try to parse, how exactly you do that, and which data you feed to the analyzer.

If you upload the sources of your protocol analyzer somewhere, we can take a look.

BTW, the tutorial has a link to an archive with a working analyzer of a made-up protocol. There are also real-life Modbus Analyzer and DF1 Analyzer; all those can be used as a reference when developing one of your own.

Hi,
Thanks for your reply, indeed i had a parsing bug in my code... fixed it and now i'm able to see parsed packets in the log .
I'm using the log.representStruct to print a stucture in the log, cool stuff !!!
Say you have a structure that contains another stucture and so on for a few levels. Is there a way to use the 'representStruct' recursively, so all inner struct will be formatted too ?

Hello Chaim,

Glad to hear you've made some good progress with your protocol analyzer!

Yes, log.representStruct should simplify and at the same time "standardize" representation of binary-based packets. And yes, you are right -- at the moment, it doesn't recursively represent all the nested structs. That's because this functionality was never required in the stock protocol analyzers in IO Ninja -- we use "layered" representation rather than "nested" because (a) this makes resulting tables more compact and easier to navigate, and (b) allows us to colorize layers using different colors. But there's nothing that prevents adding the support for nesting to log.representStruct; I think we can add that to the upcoming release.

How deep is the nesting in your case? Could you please provide definitions of the packet structs in your protocol?

Just FYI, the implementation of log.representStruct is in scripts/common/log_RepresentStruct.jnc; you can try experimenting with it by yourself until the official release.

Hi,
I tried to modify the 'scripts/common/log_RepresentStruct.jnc' to support nested structures with no success.
Can you please send me a modified version of 'scripts/common/log_RepresentStruct.jnc' that supports nested structures.
It will be a big help.
Thanks

No prob: log_RepresentStruct.jnc

Just FYI, here's the added part (special handling for struct fields):

/// ...
		if (field.m_type.m_typeKind == jnc.TypeKind.Struct) {
			representation.addHyperText($"%1%2:\t"(indent.m_sz, displayName));
			representStruct(
				representation,
				dynamic (jnc.StructType*)field.m_type,
				displayName,,
				p + field.m_offset,
				baseOffset + field.m_offset,
				0, 0, // no [+/-] header, always expanded
				indentLevel + 1
			);

			continue;
		}
/// ...

Having sub-fields collapsible is theoretically possible but won't scale well -- we have 32 total fold-flags per record, so it's better to assign those statically (rather than from a loop in a recursive function). But I think having struct fields always expanded should be good enough in most cases.

Hi Again,
Thanks for the above scripts it really works great !!!
I've presented the protocol analyzer i did to a big forum in out company, and a few question came up.

1. We would like to add this tool as part of our nightly build and validation process. Once a new build is created and deployed to a target, we'd like a way to run the IONinja via a batch file (running some command lines), load the protocol analyzer open the port and start recording the session. Is it possible to run the IONinja tool via an external API /commandline ? if so, can you send me an example how ?
2. The protocol analyzer is written in Jancy language. Is there an ability to access in Jancy APIs from DLLs created in C++ or Python ?
3. Can you activate in Jancy a batch file and do some manipulation on the bytes received ? i.e. pass the received bytes as an argument to a batch file, and get an answer from the batch file ?
   Many thanks
   Chaim

Hi again, Chaim,

From the first screenshot, I assume, you are running on Windows, correct? Then I will use Windows-style paths and declarations in the samples below.

1. Command line

Yes. You can open the necessary plugin(s) and start capturing from the command line. There are multiple ways of achieving that, and the easiest would be this.

First, configure the session the way you need -- i.e., start the main plugin (e.g., Serial Tap), attach required analyzer plugins (e.g., Modbus RTU, or your own analyzer), and configure all the necessary settings using the UI. Then, save the session somewhere (e.g., C:/Projects/playground/ioninja/my-analyzer-session/my-analyzer-session.njssn)

Now, you can start IO Ninja passing it a path to .njssn to automatically open this session. To automatically start capturing, also pass -c (i.e., connect):

ioninja -c <path-to-njssn>

2. FFI (Foreign Function Interface)

Yes. One of Jancy design goals was easy and efficient interoperability with C/C++ code. Again, multiple ways of calling external code exist, the most straightforward would be declaring an interface to your C/C++ library via the dynamiclib construct.

Let's say, you have a library called my_lib.dll with the following exported C function:

extern "C"
__declspec(dllexport) 
size_t 
my_inverse(
	void* out,
	size_t outSize,
	const void* in,
	size_t inSize
) {
	// inverse the in-buffer and write it to out-buffer
}

In your Jancy code, declare an interface to this library like this:

pragma(ThinPointers, true);

dynamiclib MyLib {
	size_t 
	my_inverse(
		void* out,
		size_t outSize,
		const void* in,
		size_t inSize
	);
}

pragma(ThinPointers, default);

Basically, function declarations are the same as in C/C++; you only have to make sure to use "thin" pointers.

Jancy pointers are "fat" -- they include extra information about region, type, etc. To achieve ABI compatibility with C/C++, we need to explicitly declare pointers as "thin", like this:

char thin* p;

If there could be many FFI declarations -- like in an interface to a C/C++ library -- it may be more convenient to implicitly treat ALL pointers as "thin" using pragma(ThinPointers, true).

Now, when you need to call your library, load and call it like this:

MyLib myLib;
myLib.open("path-to-your-lib/my_lib.dll");
myLib.lib.my_inverse(p1, size1, p2, size2);

3. Creating an external process is possible, too, of course. However, passing binary data to a process via the command line -- and especially getting an answer back -- would be much less convenient and efficient. If you need to perform some data processing in external code, I strongly recommend using the method shown above, i.e., create a C/C++ library and call it from Jancy. It's both easy to implement and as efficient as it gets in runtime.

Hope this helps!