Writing a Server Script
In this tutorial, we will show you how to listen and respond to events generated by the currently opened transport session in order to create a server which responds to incoming commands.
We will also demonstrate a very powerful technique of extracting text-based commands without explicit buffering using the Jancy reswitch statement.
Design Considerations
You may think, transport plugins are written in Jancy, and the same language is used in in-app scripting. Receiving events from transport plugins and handling those in your in-app scripts shouldn’t be that hard, right? Well, there is a catch.
Different transports behave completely differently. They generate different kinds of events with different parameters. Establishing connections works differently, as well, requires different sets of parameters, and can be instant for some transports, and lengthy for others. Reading from different transports may have different semantics, too. Bottom line is, devising a universal API which would be applicable to any transport – is nearly impossible. Attempting to do so will yield some sort of the so-called God interface, which is never a good design.
Therefore, IO Ninja is using a different approach to handling transport events in your in-app scripts. Instead of listening for IO events from transport sessions – which is very hard to design a uniform API for, your in-app scripts will be listening for new log records! The IO Ninja Log API is already uniform for all kinds of sessions (difference lies in transport-specific log record codes and record parameters). So in-app scripting is using it as a uniform way to handle events from all kinds of sessions.
Event-handling Entry Point
Your event-handling entry point will be called every time a new record is added to the log file. The declaration for this entry point looks like this:
void onLogRecord(
uint64_t timestamp,
uint64_t recordCode,
void const* p,
size_t size
)
{
// TODO: handle new log records
}
It’s not so easy to memorize the signature for this function, so we made it possible to auto-generate it with a few keystrokes. Start typing the function name (letter o), then press Ctrl + Space, and an auto-completion list will pop up. Select onLogRecord (it will likely be the only item on the list):
Hit Enter, and the skeleton for your event handler will be auto-generated.
Let’s have a quick break down of parameters passed to onLogRecord.
Parameters:
| Parameter | Description |
|---|---|
| timestamp | Timestamp of the log record (see sys.getTimestamp). |
| recordCode | Record code. May be one of log.StdRecordCode or something transport-specific. |
| p | Pointer to the block of record-code-specific parameters. |
| size | Size of record-code-specific parameters in bytes. |
Often times it’s more convenient to use a typed pointer for p instead of void const* as to avoid additional type casts. You can use any non-thin data pointer type for p, for example:
void onLogRecord(
uint64_t timestamp,
uint64_t recordCode,
char const* p, // also OK
size_t size
)
{
...
}
Basic Processing Of RX Stream
As the old saying goes, who says “A” must say “B”. Let’s write a script that will reply B for every A discovered inside the incoming (RX) data stream.
void onLogRecord( uint64_t timestamp, uint64_t recordCode, char const* p, size_t size ) { if (recordCode != log.StdRecordCode.Rx) return; // we only care about RX stream char const* end = p + size; while (p < end) { char const* a = memchr(p, 'A', end - p); if (!a) break; transmit("B"); p = a + 1; } }
To test this script, open a TCP Server on some local port (e.g. 8080), then start a TCP Connection and connect to 127.0.0.1:8080. Run this script on one side of this connection, and send some packets containing A-s from the other side. Note that you receive as many B-s back as there were A-s sent.
Buffering Commands
The formal definition of the “protocol” used in the example above is very simple – a command consists of a single character A, and it can occur anywhere in the incoming RX stream. As such, the code to process such commands is trivial.
All real-life protocols, are, of course, more complex than that. Commands typically occupy more than one byte. Also, the length of the packet is often times non-constant, but rather dependent on the contents of a particular packet.
Yet another complication to take care of is the fact that most transports – even reliable ones, such as TCP! – don’t guarantee the delivery of a packet as a whole. When you send 10 bytes on one end of a connection, you can’t expect to recv exactly 10 bytes on the other end. It can be delivered in chunks of, for example 6 and 4 bytes, or maybe 5, 3, and 2 bytes.
Bottom line – you cannot take a single log.StdRecordCode.Rx record and start analyzing its data while assuming it contains the whole packet. You have to buffer it first, and analyze it only after the whole packet has been buffered. How exactly to buffer a packet, depends on the protocol.
Plain-text Protocols
In text-based protocols, commands are usually terminated with carriage-return CR (\r) and/or line-feed LF (\n) characters. Let’s assume, CR (\r) is used as a terminator for a command. Then you can use the following boilerplate as a base for your script:
import "std_Buffer.jnc" void onLogRecord( uint64_t timestamp, uint64_t recordCode, char const* p, size_t size ) { if (recordCode != log.StdRecordCode.Rx) return; // we only care about RX stream static std.Buffer buffer; char const* end = p + size; while (p < end) { char const* cr = memchr(p, '\r', end - p); if (!cr) // not yet { buffer.append(p, end - p); break; } buffer.append(p, cr - p); // buffer the last chunk // process the buffered command // in this example, we simply write it to the log buffer.append(0); // ensure zero-termination g_logWriter.write($"command: '%s'"(buffer.m_p)); // clear buffer and move onto the next command buffer.clear(); p = cr + 1; } }
To make things more interesting, let’s assume that the command must be prefixed with STX (\x02). This is a pretty common pattern in Serial-based protocols.
import "std_Buffer.jnc" void onLogRecord( uint64_t timestamp, uint64_t recordCode, char const* p, size_t size ) { if (recordCode != log.StdRecordCode.Rx) return; // we only care about RX stream static std.Buffer buffer; static bool isCommand; char const* end = p + size; while (p < end) { if (!isCommand) // skip everything until STX { char const* stx = memchr(p, '\x02', end - p); if (!stx) break; // prepare buffer for the upcoming command p = stx + 1; isCommand = true; buffer.clear(); } char const* cr = memchr(p, '\r', end - p); if (!cr) // not yet { buffer.append(p, end - p); break; } buffer.append(p, cr - p); // buffer the last chunk // command is fully buffered, process it // in this example, we simply write it to the log buffer.append(0); // ensure zero-termination g_logWriter.write($"command: '%s'"(buffer.m_p)); // prepare for the next chunk of data p = cr + 1; isCommand = false; } }
Run this code and try sending it some data with “commands” embedded between STX and CR characters. Note that it works correctly no matter if you send those commands byte-by-byte, in chunks, or multiple commands in one block.
Binary Protocols
In binary protocols, packets are normally prefixed with a fixed-length header which carry the information about the full length of the packet. Therefore, we first need to buffer RX bytes until we have the whole header, and then buffer RX bytes until we have collected the whole packet.
import "std_Buffer.jnc" struct PacketHdr { uint16_t m_code; uint16_t m_dataSize; // followed by command-specific data (m_dataSize bytes) } void onLogRecord( uint64_t timestamp, uint64_t recordCode, char const* p, size_t size ) { if (recordCode != log.StdRecordCode.Rx) return; // we only care about RX stream static std.Buffer buffer; char const* end = p + size; while (p < end) { // first, we need to buffer the header if (buffer.m_size < sizeof(PacketHdr)) { size_t leftoverRx = end - p; size_t leftoverHdr = sizeof(PacketHdr) - buffer.m_size; if (leftoverRx < leftoverHdr) // not yet { buffer.append(p, leftoverRx); break; } buffer.append(p, leftoverHdr); p += leftoverHdr; } // the header is ready, buffer the rest of the packet PacketHdr const* hdr = (PacketHdr const*)buffer.m_p; size_t leftoverRx = end - p; size_t leftoverPacket = hdr.m_dataSize - (buffer.m_size - sizeof(PacketHdr)); if (leftoverRx < leftoverPacket) // not yet { buffer.append(p, leftoverRx); break; } buffer.append(p, leftoverPacket); // buffer the last chunk // packet is fully buffered, process it // in this example, we simply write command info to the log g_logWriter.write($"command code: %d, data: %d bytes"( hdr.m_code, hdr.m_dataSize )); // prepare for the next chunk of data buffer.clear(); p += leftoverPacket; } }
Using Jancy Regex Switches (reswitch)
The approaches shown above can be used in most cases. However, often times it’s possible to avoid the explicit buffering – and also simplify branching between possible commands.
Let’s introduce the Jancy regular expression switch, a.k.a. reswitch. Consider the example below:
void onLogRecord( uint64_t timestamp, uint64_t recordCode, char const* p, size_t size ) { if (recordCode != log.StdRecordCode.Rx) return; // we only care about RX stream static jnc.RegexState regexState( jnc.RegexStateFlags.Lexer | jnc.RegexStateFlags.Incremental ); char const* end = p + size; while (p < end || regexState.m_replayLength) { reswitch (regexState, p, end - p) { case r"about[\r\n]": transmit("Command server\r\n"); break; case r"version[\r\n]": transmit("Version 1.0.0\r\n"); break; case r"command([0-9]+)[\r\n]": transmit($"Command: %1\r\n"(regexState.m_subMatchArray[0].m_text)); break; case ".": // ignore everything else break; } p += regexState.m_consumedLength; } }
The reswitch statement takes a pointer to a jnc.RegexState variable (which holds the state of a regular expression DFA), and the pointer to the input string we are trying to analyze. In the body of reswitch you list all possible options, and if the input matches one of those, the corresponding piece of code will take control.
Note
Raw Literals
r"..." are the raw literals that keep all escape sequences intact, without escape-decoding, so that those escape sequences go directly into the regex engine. In this example, it’s not strictly necessary (regular expressions can contain CR and LF characters), but sometimes preserving the backward slash is crucial – for example, if you need to escape a special regex char such as ']'. Without the raw literals that would require escaping backward slashes as such \\\\. This works, but it yields rather unreadable code for regular expressions. With the use of raw literals you keep regular expressions neat and clean.
Lexer Mode
Let’s suppose we are matching some regular expression. We’ve discovered a match, but this match did not happen to be at the very end of the string – more characters follow. Is it still a match? Well, it depends. Sometimes, we want it to be a match, sometimes not. To choose between the two options, use jnc.RegexStateFlags.Lexer bit of the jnc.RegexState.m_flags field. When jnc.RegexState is in the lexer mode, a match followed by other characters is still a match; otherwise, it isn’t.
Incremental Mode
One of the remarkable features of reswitch is that it’s capable of processing input strings incrementally, i.e. chunk-by-chunk. The necessary buffering will be done implicitly inside jnc.RegexState. To put regex engine into the incremental mode, set the jnc.RegexStateFlags.Incremental bit of the jnc.RegexState.m_flags field.
But of course, nothing stops you from applying reswitch to a pre-buffered packet. Let’s revisit the example from the Plain-text Protocols section. After we’ve accumulated the command until a CR character, let’s use reswitch to conveniently choose an appropriate command handler:
import "std_Buffer.jnc" void onLogRecord( uint64_t timestamp, uint64_t recordCode, char const* p, size_t size ) { if (recordCode != log.StdRecordCode.Rx) return; // we only care about RX stream static std.Buffer buffer; char const* end = p + size; while (p < end) { char const* cr = memchr(p, '\r', end - p); if (!cr) // not yet { buffer.append(p, end - p); break; } buffer.append(p, cr - p); // buffer the last chunk // process the buffered command using reswitch jnc.RegexState regexState; reswitch (regexState, buffer.m_p, buffer.m_size) // run on buffer { case "about": transmit("Command server\r\n"); break; case "version": transmit("Version 1.0.0\r\n"); break; case "command([0-9]+)": transmit($"Command: %1\r\n"(regexState.m_subMatchArray[0].m_text)); break; default: transmit("Unknown command\r\n"); } // clear buffer and move onto the next command buffer.clear(); p = cr + 1; } }