dynamic layout in plain jancy

Hello!
I tried to use dylayout in "plain jancy", as it seem possible and in the changelog it is declared that replaces dynamic structs (that are interesting! but in some cases falls a bit short... I remember asking you -sometime ago- because I tried to implement a IEC103 frame generator, and it was not directly possible)

Well, I perform a quick try of the dylayout solution, and it seems even more interesting!! but I have some doubts about its use. At a very basic level

For instance, for the following code

int main()
{
        char *aa = new char[10];
        aa[0]=0x01;
        jnc.DynamicLayout layout(aa,4);

        dylayout (layout) {
                dyfield int16_t myChar;
                printf("%04d",myChar);
                // myChar = 3;          // (q1) cannot store into const location
        }
        // printf("%d",myChar);         // (q2) out of scope (expected...)
        // printf("%d",layout.myChar);  // (q2) not a member (also quite expected... but then, how?)
        return 0;
}

That works well. But,:
q1. is it possible to assign a value to a "dyfield"?
q2. is it possible to access to the "DynamicLayout" elements outside of the dylayout section?

Regards!
Josep

Hello Josep,

Happy to see you are trying to play with the new Jancy feature! Indeed, dynamic layouts are replacing dynamic structs -- which never were utilized in IO Ninja (as they were way too limited for practical use).

With this new approach, it's possible to describe pretty much any protocol or protocol stack. Please check the release announcement; there, I outlined the main problems that dynamic layouts really help with, which boils down to this:

• auto-buffering (with pause-and-resume!)
• auto-advancing the "current" pointer as we declare fields
• auto-saving of all discovered fields

To answer your questions:

q1. is it possible to assign a value to a "dyfield"?

TLDR: currently, no. In theory, yes, but that (probably) would be a misuse.

The main motivation for dynamic layouts was a simplification of binary packet parsing. Therefore, jnc.DynamicLayout expects a read-only void const* as a buffer pointer, and the Jancy compiler adds an implicit const to all fields to reinforce that.

In theory, it's possible to remove this limitation and allow passing non-const buffers to the dylayout statement (thus allowing modification of dyfield items). That shouldn't really break anything, but dynamic layouts won't really provide many benefits for the generation of packets (as opposed to parsing). The difference is that when we generate a packet, we outright know what has to be in the packet. So why not take a std.Buffer and append all the necessary blocks one by one?

q2. is it possible to access to the "DynamicLayout" elements outside of the dylayout section?

TLDR: yes and no (can enumerate all the fields, but can't reference a particular one by name or index).

Things like layout.myChar are not possible, even in theory. What if there's the branch where myChar is defined was simply skipped? Worse yet, what if myChar's type depends on the branch, like:

dylayout (layout) {
	dyfield uint8_t bitness;
	switch (bitness) {
	case 8:
		dyfield uint8_t myChar;
		break;

	case 16:
		dyfield uint16_t myChar;
		break;

	case 32:
		dyfield uint32_t myChar;
		break;

	case 64:
		dyfield uint64_t myChar;
		break;
	}
}

On the other hand, you can iterate over all the discovered fields after exiting from dylayout -- that's what IO Ninja does to represent packets in the log. To do so, you pass jnc.DynamicLayoutMode.Save to the jnc.DynamicLayout constructor and then recursively walk over sections of jnc.DynamicLayout.

Here's a rather lengthy but realistic example. To run it, simply glue all 3 code snippets below together.

First, let's define the protocol structures:

pragma(Alignment, 1);

struct EthernetHdr {
	uint8_t m_dstMac[6];
	uint8_t m_srcMac[6];
	bigendian uint16_t m_etherType;
}

struct IpHdr {
	uint8_t m_headerLength : 4;
	uint8_t m_version      : 4;
	uint8_t m_typeOfService;
	bigendian uint16_t m_totalLength;
	bigendian uint16_t m_identification;
	bigendian uint16_t m_flags          : 3;
	bigendian uint16_t m_fragmentOffset : 13;
	uint8_t m_timeToLive;
	uint8_t m_protocol;
	bigendian uint16_t m_headerChecksum;
	bigendian uint32_t m_srcAddress;
	bigendian uint32_t m_dstAddress;
}

struct TcpHdr {
	bigendian uint16_t m_srcPort;
	bigendian uint16_t m_dstPort;
	bigendian uint32_t m_seqNumber;
	bigendian uint32_t m_ackNumber;
	uint8_t m_reserved   : 4;
	uint8_t m_dataOffset : 4;
	uint8_t m_flags;
	bigendian uint16_t m_window;
	bigendian uint16_t m_checksum;
	bigendian uint16_t m_urgentData;
}

Now, here comes the main function. The dylayout part is the heart of the parser. If you want pause-and-resume, you should put it into an async coroutine -- then it will be possible to pause in the middle of parsing the packet if it's not complete yet -- and wait for more bytes. But for the TCP/IP stack, it won't make much sense, of course.

int main() {
	// a sample packet

	char packet[] =
		0x"00 1d aa 5f 9c 68 00 ad 24 90 be ae 08 00 45 00"
		0x"00 34 63 aa 40 00 80 06 00 00 c0 a8 01 79 14 bd"
		0x"ad 18 83 53 01 bb 77 02 38 0b 00 00 00 00 80 02"
		0x"fa f0 bc 0c 00 00 02 04 05 b4 01 03 03 08 01 01"
		0x"04 02";

	jnc.DynamicLayout layout(
		jnc.DynamicLayoutMode.Save, // when parsing, also save all discovered fields
		packet,
		sizeof(packet)
	);

	dylayout (layout) { // the main specification
		dyfield EthernetHdr hdr;

		switch (hdr.m_etherType) {
		case 0x0800: // IP4
			dyfield IpHdr ipHdr;
			ipHdr.m_protocol = 6;

			if (ipHdr.m_headerLength * 4 > sizeof(IpHdr)) // have options
				dyfield uint8_t options[sizeof(IpHdr) - ipHdr.m_headerLength * 4];

			switch (ipHdr.m_protocol) {
			case 6: // TCP
				dyfield TcpHdr tcpHdr;
				break;

			case 17: // UDP
			case 1: // ICMP
			// etc
			}
			break;

		case 0x86dd: // IPv6
		case 0x0806: // ARP
		// etc
		}
	}

	printGroup(packet, layout);
	return 0;
}

Finally, here's how to do a recursive walk across all discovered items. A more sophisticated version of such walker could be found in scripts/common/log_RepresentDynamicLayout.jnc (it's used to render dynamic layouts in the log with respect for color, format specifier, display name, and other attributes):

string_t g_indentStep = "    ";

void printGroup(
	void const* p,
	jnc.DynamicSectionGroup* group,
	string_t indent = ""
) {
	for (size_t i = 0; i < group.m_sectionCount; i++) {
		jnc.DynamicSection* section = group.m_sectionArray[i];
		switch (section.m_sectionKind) {
		case jnc.DynamicSectionKind.Array:
			printf("%08x%s %s %s[%d]\n", section.m_offset, indent, section.m_type.m_typeString, section.m_decl.m_name, section.m_elementCount);
			break;

		case jnc.DynamicSectionKind.Struct:
			jnc.StructType* type = dynamic (jnc.StructType*)section.m_type;
			printFields(p, section.m_offset, type, indent);
			break;

		case jnc.DynamicSectionKind.Group:
			printf("%08x%s %s {\n", section.m_offset, indent, section.m_decl.m_name);
			printGroup(p, section, indent + g_indentStep);
			printf("%s}\n", indent);
			break;
		}
	}
}

void printFields(
	void const* p,
	size_t baseOffset, // struct field offsets are relative to the beginning of the struct, so we need base offset
	jnc.StructType* type,
	string_t indent
) {
	for (size_t i = 0; i < type.m_fieldCount; i++) {
		jnc.Field* field = type.m_fieldArray[i];
		size_t offset = baseOffset + field.m_offset;
		printf("%08x%s %s %s", offset, indent, field.m_type.m_typeString, field.m_name);

		if (field.m_type.m_typeKind != jnc.TypeKind.Struct)
			printf(" = %s\n", field.getValueString(p + offset));
		else {
			printf("\n");
			printFields(p, offset, dynamic (jnc.StructType*)field.m_type, indent + g_indentStep);
		}
	}
}

But what if you want to access a particular field instead of walking across all fields? Then you need to access it within dylayout, from the branch where this field is visible! Otherwise, the field you try to access may be missing or be of the wrong type.

A short summary.

1. dynamic layouts are for parsing binary packets; using them for packet generation is possible in theory, but gives (almost) no benefits compared to standard methods.
2. a recursive walk across all discovered fields is possible -- IO Ninja does just that!
3. accessing dynamic fields should be done from within dylayout, while we see the declaration and know the field is there.

I know, it's a lengthy reply, but hope this makes sense. Feel free to follow up with any questions or suggestions!

A short follow-up after thinking more about q1.

In my previous write-up, I used the word "packets" when talking about binary blobs that dynamic layouts work with. But of course, those could be any binary objects — disk files, disks themselves, shared memory, etc.

Indeed, with packets, we usually generate the whole thing from scratch—and using dynamic layouts here doesn't offer much.

But if we think about objects like files or disks, it makes perfect sense to allow modification alongside parsing. Something like (1) locate a specific field inside a file, (2) modify this field, (3) proceed to the next one.

So yeah, I think we should remove the forced const on dyfield declarations. We still need to somehow preserve const-correctness for the parse-only cases, though.

One way would be to introduce an auxiliary class jnc.MutableDynamicLayout, which would take non-const pointers; when the dylayout argument is jnc.MutableDynamicLayout, the Jancy compiler won't add implicit const to dyfield declarations.

Thoughts?

P.S. Moved the topic to General Discussion

Hello @vladimir
Thank you very much for your detailed explanation, as always it is very illustrative!!
Ok! More or less, I have the feeling that it was going to be as you comment, that the main purpose and utility of this construction is for parsing (in fact, my first impression is that it should be used inside the ioninja app (for ioninja scripts), and could not be used in "plain jancy", I was happy when I see that in fact could be used "alone").
As you said, I was thinking of using it for parsing and generating frames (IEC60870-5-103 frames), but, as you point out, it is perhaps not really very useful when constructing a frame.
Again, thank you!
Regards!
Josep

@jose-marro

I was thinking of using it for parsing and generating frames (IEC60870-5-103 frames)...

For parsing frames, dylayout would be a perfect tool.

For generating frames programmatically, dynamic layouts won't add much extra convenience (and dyfields are currently all const anyway).

But! One thing I didn't mention above is that dynamic layouts are also perfect for creating packet templates!

After defining a dynamic layout "specification" for a protocol, you will be able to conveniently build packets for this protocol in the Property Grid on the Hex Transmit pane in IO Ninja. Set enumeration fields via drop-down lists, set bits in bitfields with check-boxes, have big-endian automatically converted for you, etc. If a protocol uses checksums, you can define methods for automatic calculation of those checksums before transmission.

All in all, it's an awesome tool for generating and sending out test packets! If you didn't see it, please check it out -- it could be just what you are looking for.

Hello! About your last post (2-jul), I will try it!
And about the previous one (1-jul; 04:16), sorry but I did not see it and I write my post -later- without been aware of it (I did it very quickly, and I was a bit confused at that moment because I spent some time finding the the thread and wandering if all was a dream (I am getting older, and I am not sure of nothing ). All in all, my post was not very coherent with what you said about having "mutable" dynamic layouts, well, I think that this could be useful, and if it is something possible without a really big difficulties/chages, I think that could be eventually useful in some cases, as you point out.
Regards!
Josep