CLI use (ioninja - serial tap)

Hello.
I'm a happy owner of a (tibbo) serial tap and ioninja SW.
Now, I want to use them in order to capture data, in a remote location, unattended and with bad communications.
I am going to use a Rasperry (Arch Arm) with a GPRS modem (not sure if 3G is available).
I do not need the GUI (and it could be a problem if I have to use it!).
Ideally, I would like to be able to save a log, and download it from time to time. I've seen that there is an executable "ioninja-server" that seems that could do the trick, but I have not found detailed documentation (just the "-h", and a short reference in your web).

So the question are: Is there any way to do what I want? With this "ioninja-server"? how?

Thank you very much!

Apologies for the delayed reply... Still, better late than never

Alas, currently, there's no support for remote monitoring via Serial Taps -- fiddling with ioninja-server won't cut it... However, it's on our immediate TODO list; we are working on a major update scheduled for the beginning of 2022 which -- among many other things -- will allow controlling Serial Taps and other IO Ninja hardware sniffers connected to remote ARM-based Linux boxes (such as Raspberry Pis).

If waiting for the update is not an option, I suggest writing your own program which would read from a Serial Tap using libusb (or any other low-level USB framework available in the programming language of your choice) and dump the captured data into a file.

The low-level details of USB communication with a Serial Tap (USB VIDs/PIDs, IN/OUT endpoints, packet formats, etc) can be checked in the open-source implementation of the official Serial Tap plugin (see file scripts/plugins/SerialTap/SerialTapSession.jnc). If you are familiar with C-family languages (C, Java, JavaScript, etc) you should be able to read and understand pretty much everything. And of course, feel free to ask any implementation-related questions here.

Hope this clarifies the issue.

Hello. Do not worry about the delay, we are all very busy
It will be good to have it. In the meantime, libusb could be an option. I used to program in C -many time ago- and I will try (perhaps, I will eventually bother you a bit, with some questions).
Thank your! Regards!

Hello. Thank you! I've seen that the recent release includes a tool in order to achieve what I wanted to do.
Just a improvement for future releases, it would be great if the tool would save a log, in ".njlog" format, part of my problem is the lack of communications, so keep a ssh session alive it is not an option (I saw that it is possible save "pcap" for ethernet captures, but -obviously- we are talking about serial).
Also, I checked the "write to file" option, and I have two comments:

1. I saw that as "sidecar" of the sniffed data there are some other data (4 bytes ?), I suspect it is info about the data origen, perhaps used by the "ioninja" SW in order to identify it. Is there any place where we can check the meaning of these bytes?
2. I suspect that inside this "sidecar" data there is no timestamps. The HW probably does not timestamp its raw data... but it would be quite necessary that the output files writen by the "ioninja-hwc" do include timestamps.
   Regards!!

Hello Jose,

Once again, apologies for the delayed reply

Yes, in IO Ninja 5.1 we added a CLI tool (ioninja-hwc) for direct communication with Serial Tap and other IO Ninja hardware. The main purpose of this tool was to network-enable the IO Ninja Taps (i.e., to access those over SSH). As such, the output from this tool must satisfy certain requirements (e.g., reliably distinguish between ioninja-hwc messages and SSH server errors, like ioninja-hwc not found). The .njlog format is not very suitable for that, that's why we didn't use it by default (we actually used the protocol from our test fixture for Serial and other Taps).

This said, we can of course add .njlog output as an extra feature -- just like we have .pcap output. We probably will do that in one of the upcoming releases.

1. Regarding the format of the output -- since all the scripts in IO Ninja are open-source, you can check the sources of decoder in ioninja/scripts/common/io_HwcProto.jnc. This file contains file structure definitions, packet type code constants, and the decoder itself.

2. You are right, there are no timestamps in this protocol. Serial Tap doesn't timestamp individual bytes (Serial Tap basically is a dual-channel USB-to-UART). We can, of course, add timestamps inside ioninja-hwc, but in case of SSH communications, it's not that much different from timestamping on the SSH client side (as we do now). And after we support the .njlog format for file output, adding timestamps to the protocol most likely will not be necessary.

Let me know what you think!

Hello Vladimir,
No worry about the delay and thank you !
Ok! I think that I will be able to use the current output, and it could solve my problems (at least one of them).
Eventually, if you could add the function of save as ".njlog", it would be great.
As a piece of info, that could help someone, I am currently trying "xpra" (https://xpra.org/), it is a kind of X server that allows remote access. This allows run ioninja remotely (and keep it alive permanently) and "attach" me to it from time to time remotely.
Regards!
Josep

Yes, we plan to add the .njlog output soon, so that would solve many problems.

Running a remote X-server is an option, yes. It actually was the only option for running Serial Tap remotely before the release of ioninja-hwc. Now that we have ioninja-hwc, it should be the preferred approach in most cases.

We also consider extending the Serial over SSH (and other remote tap plugins) with an option of keeping the ioninja-hwc process persistent across SSH reconnects (via screen or some other terminal manager).

Hi there, I think this is the most related to what I am doing currently. I have the ioninja software and the serial tap and I've been trying to log files through the ioninja-hwc. My issue currently is trying to set up the output file format/extension to get a usable file. Likely I just don't understand what the "write to file option" should be setup as. currently I just have it running as: "-out=logfile.txt" but don't really get anything useful from that.

My end goal is to generate a log file (like an .njlog if possible) through terminal alone. If there's a different way to get a log file without needing to open the IDE I'd really appreciate the help

Hello,

Apologies for the delayed response. Support for njlog output directly from ioninja-hwc is on our TODO list (it was already requested by users). For the time being, you need to post-process the raw output of ioninja-hwc using any scripting language of your choice.

The protocol of communication between ioninja-hwc and plugins such as Serial over SSH or Serial Tap over SSH is not currently documented, but the decoding process and all relevant constants and data structures can be looked up in the sources of those plugins (remember, all IO Ninja plugins are open-sourced and available at ioninja/scripts/plugins).

In particular, all the relevant constants and packet structures for the ioninja-hwc protocol are contained in ioninja/scripts/common/io_HwcProto.jnc. The output file generated by ioninja-hwc is basically a sequence of "out" messages of this protocol. Each message starts with HwcMsgHdr followed by extra data block, the meaning of which depends on the message code. For example, HwcMsgCode.Rx is followed by the received bytes, HwcMsgCode.SerialTapCtsDsr is followed by HwcSerialStatusLines flags, etc.

Try decoding the output of ioninja-hwc using any language of your choice, and feel free to let me know if you run into a stumbling block or have any other questions.

Hello Vladimir!
Just a few words in order to say that I've already saw that this feature is included in the last release. In fact, I've already used it and it works great! In fact, at this moment -since about 10 days-, it is working in the setup I commented in my first mail (a raspberry, with a serial tap, in a remote location with very bad commications (I just login in order to see that it is working)). I find it really useful!
Thank you very much!
Regards
Josep

Hello Jose,

Thank you for your feedback, Jose! Glad to hear the new feature worked well for you!

Adding the support for .njlog output was the right thing to do; thanks for bringing that up originally and with that, adjusting our TODO list priority