Installation of Tibbo Device Monitor on Linux
Tibbo Device Monitor service provides device monitoring capabilities for IO Ninja. It’s the engine behind the Serial Monitor plugin, Named Pipe Monitor plugin and such. Monitoring of device IO on Linux is currently performed by hooking the
file_operations structure of the corresponding device, but the hooking details and techniques may change in the future.
Download the latest
tdevmon-linux-<version>-<arch>.tar.xz tarball from https://ioninja.com/downloads
Unpacking the archive yields the following directory structure:
bin/ include/ tdevmon/ lib/ cmake/ tdevmon/ share/ tdevmon/ license/ src/ lkm/
The core of Tibbo Device Monitor service on Linux consists of the following two main components:
- Service management application
- Loadable kernel module
tdevmon executable resides in
bin/ directory and can be executed directly. Run
./tdevmon --help for available options.
Starting with IO Ninja version 3.10.0, there is support for remote monitoring over SSH (e.g. the Serial Monitor over SSH plugin). In order for these plugins to work, the
tdevmon executable must be directly accessible via
PATH – at least, for the account used to login over SSH. Probably, the simplest solution is to create a symbolic link to the
tdevmon executable inside a standard program directory such as
$ sudo ln -s $TDEVMON_DIR/bin/tdevmon /usr/local/bin/tdevmon
The loadable kernel module (LKM)
tdevmon.ko cannot be distributed as a pre-compiled binary (well, technically speaking, it can, but that would be a wrong thing to do). Instead, it must be built against the exact version of Linux kernel running on your machine. The sources of
tdevmon.ko can be found in
First of all, you need to install the system headers required to build Linux kernel modules. The actual way of doing so depends on the Linux distribution; usually, the package with the required headers is called
linux-headers, but it may vary.
On Ubuntu and most other Debian-based distributions, run:
$ sudo apt-get install linux-headers-$(uname -r)
On Raspbian (the default distro on Raspberry Pi) the name of the package is different:
$ sudo apt-get install raspberrypi-kernel-headers
On Arch Linux, run:
$ sudo pacman -S linux-headers
… and so on. Refer to your distribution reference on building Linux kernel modules.
After that, navigate to the
share/tdevmon/src/lkm folder and inside it simply run
$ cd <tdevmon-dir>/share/tdevmon/src/lkm $ make make -C /lib/modules/4.12.3-1-ARCH/build/ M=<tdevmon-dir>/share/devmon/src/lkm modules make: Entering directory '/usr/lib/modules/4.12.3-1-ARCH/build' CC [M] <tdevmon-dir>/share/devmon/src/lkm/module.o CC [M] <tdevmon-dir>/share/devmon/src/lkm/Device.o CC [M] <tdevmon-dir>/share/devmon/src/lkm/Hook.o CC [M] <tdevmon-dir>/share/devmon/src/lkm/Connection.o CC [M] <tdevmon-dir>/share/devmon/src/lkm/HashTable.o CC [M] <tdevmon-dir>/share/devmon/src/lkm/ScatterGather.o CC [M] <tdevmon-dir>/share/devmon/src/lkm/lkmUtils.o LD [M] <tdevmon-dir>/share/devmon/src/lkm/tdevmon.o Building modules, stage 2. MODPOST 1 modules CC <tdevmon-dir>/share/devmon/src/lkm/tdevmon.mod.o LD [M] <tdevmon-dir>/share/devmon/src/lkm/tdevmon.ko make: Leaving directory '/usr/lib/modules/4.12.3-1-ARCH/build'
tdevmon.ko is built.
Sometimes, even after a successful installation of the
linux-headers package, attempts to build
tdevmon still fail with error messages similar to the ones below:
$ make make -C /lib/modules/4.12.3-1-ARCH/build/ M=<tdevmon-dir>/share/devmon/src/lkm modules make: *** /lib/modules/5.9.1-arch1-1/build/: No such file or directory. Stop.
This normally indicates that the
linux-headers package and the Linux kernel are out-of-sync (most likely,
linux-headers is newer than the kernel). Make sure that the two match (by upgrading the kernel, or by downgrading
Loading tdevmon.ko manually
tdevmon.ko into Linux kernel manually, run from within the same folder:
sudo insmod tdevmon.ko
This will create a device called
/dev/tdevmon which can be used to communicate with the kernel module and install or remove hooks on selected devices.
You have to realize that providing device IO monitoring capabilities introduces certain security risks. Therefore, by default the
/dev/tdevmon device only allows
root access; non-superusers simply would not be able to open it. During debugging, however, it may be desirable to alleviate access limitations. You can do so with either changing permissions/ACLs of the
/dev/tdevmon post-creation, or by overriding the
permissions parameter while loading
sudo insmod tdevmon.ko permissions=0666
The command above will allow read-write access to
/dev/tdevmon for everybody.
tdevmon.ko is properly loaded and configured, please navigate to
bin/ and run:
$ ./tdevmon --stats tdevmon.ko stats: Total devices: 0 Connections: 0
Loading tdevmon.ko at boot
To make a kernel module loaded at boot, you need to add it to the
Copy or symlink the kernel module into
lib/modules/$(uname -r)/kernel, e.g:
depmodto add an entry into the modprobe dependency database:
After that you should be able to find a line in /lib/modules/4.14.71-v7+/modules.dep:
/etc/modulesfile and add this line:
This will make tdevmon auto-loaded at boot.
(OPTIONAL) Specify permissions to allow non-sudo access.
Create a file /etc/modprobe.d/tdevmon.conf and add this line:
options tdevmon permissions=0666
This is not strictly essential but highly recommended if you plan to access it with the SSH Serial Monitor plugin (normally you don’t login over SSH with the root account).
After reboot you shall see the /dev/tdevmon device with permissions 0666
tdevmon.ko has been successfully loaded, you can start monitoring device IO either by using IO Ninja plugins (such as Serial Monitor) or by running:
$ ./tdevmon --monitor <device-name> # for example: $ ./tdevmon --monitor /dev/ttyUSB0
By default, all IOCTL arguments will be displayed just as numbers – the POSIX standard doesn’t specify the size of data pointed to by an IOCTL pointer argument, so
tdevmon can’t uniformly extract and show the IOCTL argument data in a way which would work for all devices.
If you do need to see the contents of IOCTL pointer arguments for a particular device (e.g.
termios struct when monitoring serial ports), please use the
--ioctl <code>:<size> switch, e.g.:
$ ./tdevmon --monitor /dev/ttyUSB0 --ioctl 0x5402:60
This will display a hex dump of the
termios struct passed as the argument for
When you start monitoring device IO,
tdevmon.ko installs hooks on the specified devices. There is a fundamental problem with any hooks, and that is the so-called hook removal problem. You can only safely remove the hook which was installed the latest – in other words, in the LIFO order.
Imagine some other module is using the same technique and it has re-hooked the device after
tdevmon.ko has done so. This new module is now holding pointers to the code in
tdevmon.ko. It means, that we can’t simply restore the original pointers and unload
tdevmon.ko anymore – this new module can call us any time! Moreover, during its removal, this new module will restore the pointers inside the hooked
file_operation struct to point back to
tdevmon.ko – thinking it’s the original!
Long story short, removing hooks is much harder than installing them. The approach to the hook removal problem used in Tibbo Device Monitor is this. Unloading is split into the two stages. First, you need to stop the
tdevmon.ko. During this stage,
tdevmon.ko will stop dispatching notifications, stop accepting any new connections from applications willing to monitor device IO, and, most importantly, try to unhook all the devices. Naturally, this operation can fail – if
tdevmon.ko detects a hook override by some other module. Only when – and if! – stop stage succeeded, you can safely unload
$ ./tdevmon --stop $ sudo rmmod tdevmon.ko
Note that after a successful stop
tdevmon.ko no longer serves any further requests – they will fail with
EBADFD error (i.e. invalid state).
It’s also possible to try and unhook a particular device without stopping
./tdevmon --unhook /dev/ttyUSB0
When all the hooks are removed, it’s safe to issue an
rmmod. Note that if there is at least one hook left,
rmmod will fail with:
$ sudo rmmod tdevmon.ko rmmod: ERROR: Module tdevmon is in use