tdevmon - segmentation fault

Hello.
I am trying to use tdevmon, in a Raspberry PI4, with ArchArm linux.
I followed the instructions detailed in the installation guide (they are great!), and I have no problem in build the kernel module. All was ok. Then I install the kernel module -all ok-. But when I tried to use the tdevmon.bin, then the first try give me a "segmentation fault" message, and then, doing the same again, it does not give the error message, but it seems that it is not working ok, as it does not capture any activity.

I put some images showing it.
9e7651f9-7ef3-4c0b-ad85-d3f3f97c8f03-image.png
78cfb6b1-1a85-4c9a-a0bc-1b64b54313d4-image.png

Likewise -and not unexpectedly- , if I then try to use "ioninja", I am able to start the capture, but it does not capture any communications
81cafc58-cde5-4550-b5ca-9c1a86f55ed4-image.png

do you think I could do something?

I take advantage of this message to:

  • ask: once solved the issue... does tdevmon save the data in njlog format?
  • comment: well, as you see I like ArchArm... they seems to prefer the AArch64, and even for some boards, they no provide the 32 bits version... I do not know how much workload is, but if it not much, perhaps you could consider to release the Arm version also for 64 bit architecture (just the tarball)

Thank you! and regards

Hello Jose,

Finally, found some time to check the issue -- and guess what? Turns out, it was already solved before -- but the download link was reverted back to the previous version for some reason. In any case, please take the patched version v3.3.9-a and try that one; it should work.

Now, to your other questions.

does tdevmon save the data in njlog format?

No, it doesn't, but that's because it's not possible to do in the general case. Thing is, the format of IOCTL parameters is device-specific, so the job of decoding IOCTLs and writing proper .njlog records lies on a particular plugin (such as Serial Monitor, Pipe Monitor, Mailslot Monitor, the upcoming USB Monitor, etc). Yes, it's theoretically possible to emit a severely trimmed version of .njlog that would contain read-s and write-s only (which are universal), but that's a serious limitation, so I'm not really sure such a feature would be useful.

perhaps you could consider to release the Arm version also for 64 bit architecture (just the tarball)

Yes, that definitely can and should be done. As a matter of fact, just a day ago @bob suggested porting tdevmon to AArch64 so we might get something in this department very soon.

Hello Vladimir!
sorry for the delay, I tried few day ago it but I would like to recheck.

@vladimir said in tdevmon - segmentation fault:

Hello Jose,

Finally, found some time to check the issue -- and guess what? Turns out, it was already solved before -- but the download link was reverted back to the previous version for some reason. In any case, please take the patched version v3.3.9-a and try that one; it should work.

Well, I am sorry but I have to say that this patched version behaves to me exactly as the previous one, so I get the same "segmentation fault" error. If I could provide any other information that could be useful to you, please let me know.

Regarding the njlog forma

Now, to your other questions.

does tdevmon save the data in njlog format?

No, it doesn't, but that's because it's not possible to do in the general case. Thing is, the format of IOCTL parameters is device-specific, so the job of decoding IOCTLs and writing proper .njlog records lies on a particular plugin (such as Serial Monitor, Pipe Monitor, Mailslot Monitor, the upcoming USB Monitor, etc). Yes, it's theoretically possible to emit a severely trimmed version of .njlog that would contain read-s and write-s only (which are universal), but that's a serious limitation, so I'm not really sure such a feature would be useful.

I get the point. A pity, but you are absolutely right.

perhaps you could consider to release the Arm version also for 64 bit architecture (just the tarball)

Yes, that definitely can and should be done. As a matter of fact, just a day ago @bob suggested porting tdevmon to AArch64 so we might get something in this department very soon.

Thank you!!
Regards

Hello Jose,

Just so that we are on the same page -- the segmentation fault happens in the kernel module, not in the app. So, after unpacking the 3.3.9-a patch, you need to rebuild the tdevmon.ko module from the updated sources -- not simply run the new bin/tdevmon.

If a segmentation fault occurred earlier, you might be unable to unload tdevmon.ko, so it's better to reboot to ensure the old module is gone.

If the segmentation fault still occurs, could you please try v3.3.8 from the archive:

https://tibbo.com/downloads/archive/tdevmon/

The regression under suspicion was introduced in v3.3.9 (and -- presumably -- fixed in v3.3.9-a). If v3.3.8 still crashes, it might be for a completely unrelated reason. Once again, the important thing is to ensure you rebuild and reload tdevmon.ko

If the segmentation fault still occurs, I will investigate further. Originally, I tried it on Raspberry Pi 3 with the latest Raspbian with a modern Linux-5.15 kernel, and it worked; I might need to get the exact version of the kernel (maybe, even the exact distro).

Hello Vladimir!
yes, I had had built the 3.3.9_a tdevmon.ko, and it also happens.

I've tried the 3.3.8 and first I was not able to compile it
f931311a-6de0-4978-ac57-504da3912852-image.png

I quickly look at lkmUtils.c, and I see there a difference between 3.3.8 and 3.3.9_a.
5760f5dc-841f-44a6-8b6a-db7950f84a0f-image.png
I just copy this section from 3.3.9_a into 3.3.8, and then I could make it. But the when I run the tdevmon.bin (obviously, after installing the built module), I had got the same "Segmentation Fault" error. Hope it helps 🙂

Regards... and have a Happy Xmas!
Josep

Hello Jose,

Merry Christmas and Happy New Year to you, as well!

I installed ArchLinux for ARM, and you were right. It does segfault over there -- while it doesn't on Raspbian with nearly the same kernel version.

After an investigation it turned out that the difference is that the kernel for ArchLinux is built with CONFIG_ARM_LPAE (support for Large Physical Address Extension) while Raspbian kernel is built without one. I remember testing write protection removal on ARM LPAE kernels, but that was with a relatively old kernel. In recent kernels, a slight modification in PMD (Page Middle Directory) flags is required.

Keeping the write protection removal code compatible with new Linux kernels is the whack-a-mole game. It works, then a new kernel comes out, and it stops compiling or working; you bring it up to speed, and it works again until the next breaking change in the kernel...

Anyway, please try tdevmon-3.3.10 (https://tibbo.com/downloads/archive/tdevmon/tdevmon-3.3.10/tdevmon-3.3.10-linux-arm32.tar.xz) and let me know if it works for you.

Hello Vladimir,
I can confirm that it works in my setup (at least until next kernel-mole shows up).
Thank you and regards!

@vladimir
Hello! I saw that you just release a new version of ioninja, I am eager to try it. Thank you!
... but (there is always a "but" 🙂 ) I saw that there is no the Aarch64 version. It would be very useful to me to have the Aarch64 version of "tdevmon" (as ASAP as possible). Could you be able to provide it, at least as "pre-release version"? It would be really useful!
Regards!

Sorry! I said "tdevmon", but what I need is the ioninja-hwc !!

Hello Jose,

Yes, ioninja-5.3.0 is out; it's quite a big release. Please try it and let me know what you think of the new form UI!

As for Aarch64 builds -- yeah, we still don't have those, sorry. I remember @bob was working on porting tdevmon to Aarch64, and I extracted and prepared a repo for this project, but I didn't receive any pull requests since, so I guess it didn't quite work out 😞

Making an Aarch64 build for ioninja-hwc should be relatively easy. However, it should be possible to run an ARM build of ioninja-hwc on Aarch64 boxes directly, no?

In any case, I guess, it's time for us to dedicate some actual efforts to the Aarch64 Linux builds of IO Ninja & tdevmon 🙂

Hello Vladimir,
The problem is that it seems that it is not possible to run the 32 bit version on the box if the linux version is 64 bits.
For example, the Arch Arm SO for Raspberry 4 is available in 32 and 64 bits. So I could run ioninja-hwc if the SO is the 32 bits version, but it does not run in the 64 bit SO version. The problem is that for other boxes (eg, I am testing a Rock64) there is only available the 64 bits version of ArchArmLinux SO... An it seems that it is difficult to find 32 bits SO for ARMv8 plataforms. I tried to find an alternative to ArchArm for this Rock64 SBC, and look into AOSC and Armbian, but I think that they are also 64 bits versions.
The reason I am using a Rock64 is basically availability, I tried to find Raspberries, but they are out of stock.
Regards!

ops... when I wrote "SO" I meant "OS" ("linux")... I'm a bit dyslexic (and I my language so it goes this acronym), and I did it really quickly

Understood. We will try to add Aarch64 builds in the near future.

@vladimir 👍 thank you!

Hello again Jose!

It took me a while to address the write protection removal issue on aarch64, but I finally got tdevmon working on aarch64; there's also an experimental build of IO Ninja for aarch64. Would you be willing to test those? Which is the most important module for you, ioninja-hwc or tdevmon?

Hello Vladimir!
Thank you! Of course, I will be glad to test them!
For me, it is more important the ioninja-hwc, as what I want to do is deploy a device with a serial tap attached to it.
Regards

Hello Vladimir,
I've done a quick tests.
ioninja-hwc: I have tested it with a (1) serial-tap, and it worked ok.
tdevmon: I found one error compiling the kernel module. It gave a "incompatible-pointer-type" error. The same kind we saw before, in this same email chain, perhaps it is something related.

Clipboard 3.png

(I've checked that the linux-headers and the kernel version match)
Clipboard 4.png

Just fyi, also I tried the "GUI" ioninja, and it gave me an error when "JITting", e.g., going to use the "serial tap" plug-in... but perhaps it is normal and expected, as we have just been talking about the "ioninja-hwc" (and "tdevmon")

Clipboard 1.png
Clipboard 2.png

Regards!

Hmm. Everything is working well on the fully updated Raspbian Aarch64; I installed ArchLinux ARM Aarch64 -- and everything fails there the same way you describe. Fixing the compilation error for tdevmon was easy, but that didn't quite cut it -- it looks like write protection removal needs extra polishing for the newer kernels.

Ok! Do not worry! In the mean time, if I need it, I would -with some care- use the ioninja-hwc as it seems to work well. If I found anything unexpected, I will let you know. Regards and thank you!