Installation of Tibbo Device Monitor on Linux

Preface

Tibbo Device Monitor service provides device monitoring capabilities for IO Ninja. It’s the engine behind the Serial Monitor plugin, Named Pipe Monitor plugin and such. Monitoring of device IO on Linux is currently performed by hooking the file_operations structure of the corresponding device, but the hooking details and techniques may change in the future.

Installation

Download the latest tdevmon-linux-<version>-<arch>.tar.xz tarball from https://ioninja.com/downloads

Unpacking the archive yields the following directory structure:

bin/
include/
    tdevmon/
lib/
    cmake/
        tdevmon/
share/
    tdevmon/
        license/
        src/
            lkm/

The core of Tibbo Device Monitor service on Linux consists of the following two main components:

  • Service management application tdevmon
  • Loadable kernel module tdevmon.ko

The tdevmon executable resides in bin/ directory and can be executed directly. Run ./tdevmon --help for available options.

Note

Starting with IO Ninja version 3.10.0, there is support for remote monitoring over SSH (e.g. the Serial Monitor over SSH plugin). In order for these plugins to work, the tdevmon executable must be directly accessible via PATH – at least, for the account used to login over SSH. Probably, the simplest solution is to create a symbolic link to the tdevmon executable inside a standard program directory such as /usr/local/bin:

$ sudo ln -s $TDEVMON_DIR/bin/tdevmon /usr/local/bin/tdevmon

The loadable kernel module (LKM) tdevmon.ko cannot be distributed as a pre-compiled binary (well, technically speaking, it can, but that would be a wrong thing to do). Instead, it must be built against the exact version of Linux kernel running on your machine. The sources of tdevmon.ko can be found in share/tdevmon/src/lkm.

Building tdevmon.ko

First of all, you need to install the system headers required to build Linux kernel modules. The actual way of doing so depends on the Linux distribution; usually, the package with the required headers is called linux-headers, but it may vary.

On Ubuntu and most other Debian-based distributions, run:

$ sudo apt-get install linux-headers-$(uname -r)

On Raspbian (the default distro on Raspberry Pi) the name of the package is different:

$ sudo apt-get install raspberrypi-kernel-headers

On Arch Linux, run:

$ sudo pacman -S linux-headers

… and so on. Refer to your distribution reference on building Linux kernel modules.

After that, navigate to the share/tdevmon/src/lkm folder and inside it simply run make:

$ cd <tdevmon-dir>/share/tdevmon/src/lkm
$ make
make -C /lib/modules/4.12.3-1-ARCH/build/ M=<tdevmon-dir>/share/devmon/src/lkm modules
make[1]: Entering directory '/usr/lib/modules/4.12.3-1-ARCH/build'
    CC [M]  <tdevmon-dir>/share/devmon/src/lkm/module.o
    CC [M]  <tdevmon-dir>/share/devmon/src/lkm/Device.o
    CC [M]  <tdevmon-dir>/share/devmon/src/lkm/Hook.o
    CC [M]  <tdevmon-dir>/share/devmon/src/lkm/Connection.o
    CC [M]  <tdevmon-dir>/share/devmon/src/lkm/HashTable.o
    CC [M]  <tdevmon-dir>/share/devmon/src/lkm/ScatterGather.o
    CC [M]  <tdevmon-dir>/share/devmon/src/lkm/lkmUtils.o
    LD [M]  <tdevmon-dir>/share/devmon/src/lkm/tdevmon.o
    Building modules, stage 2.
    MODPOST 1 modules
    CC      <tdevmon-dir>/share/devmon/src/lkm/tdevmon.mod.o
    LD [M]  <tdevmon-dir>/share/devmon/src/lkm/tdevmon.ko
make[1]: Leaving directory '/usr/lib/modules/4.12.3-1-ARCH/build'

Congratulations! tdevmon.ko is built.

Note

Sometimes, even after a successful installation of the linux-headers package, attempts to build tdevmon still fail with error messages similar to the ones below:

$ make
make -C /lib/modules/4.12.3-1-ARCH/build/ M=<tdevmon-dir>/share/devmon/src/lkm modules
make[1]: *** /lib/modules/5.9.1-arch1-1/build/: No such file or directory.  Stop.

This normally indicates that the linux-headers package and the Linux kernel are out-of-sync (most likely, linux-headers is newer than the kernel). Make sure that the two match (by upgrading the kernel, or by downgrading linux-headers).

Loading tdevmon.ko manually

To load tdevmon.ko into Linux kernel manually, run from within the same folder:

sudo insmod tdevmon.ko

This will create a device called /dev/tdevmon which can be used to communicate with the kernel module and install or remove hooks on selected devices.

Permissions

You have to realize that providing device IO monitoring capabilities introduces certain security risks. Therefore, by default the /dev/tdevmon device only allows root access; non-superusers simply would not be able to open it. During debugging, however, it may be desirable to alleviate access limitations. You can do so with either changing permissions/ACLs of the /dev/tdevmon post-creation, or by overriding the permissions parameter while loading tdevmon.ko:

sudo insmod tdevmon.ko permissions=0666

The command above will allow read-write access to /dev/tdevmon for everybody.

To ensure tdevmon.ko is properly loaded and configured, please navigate to bin/ and run:

$ ./tdevmon --stats
tdevmon.ko stats:

Total devices:    0
Connections:      0

Loading tdevmon.ko at boot

To make a kernel module loaded at boot, you need to add it to the modprobe database.

  1. Copy or symlink the kernel module into lib/modules/$(uname -r)/kernel, e.g:

    /lib/modules/4.14.71-v7+/kernel/misc/tdevmon.ko

  2. Run depmod to add an entry into the modprobe dependency database:

    $ depmod

    After that you should be able to find a line in /lib/modules/4.14.71-v7+/modules.dep:

    kernel/misc/tdevmon.ko

  3. Edit the /etc/modules file and add this line:

    tdevmon

This will make tdevmon auto-loaded at boot.

  1. (OPTIONAL) Specify permissions to allow non-sudo access.

    Create a file /etc/modprobe.d/tdevmon.conf and add this line:

    options tdevmon permissions=0666

    This is not strictly essential but highly recommended if you plan to access it with the SSH Serial Monitor plugin (normally you don’t login over SSH with the root account).

  2. Reboot.

    After reboot you shall see the /dev/tdevmon device with permissions 0666

Monitoring devices

After tdevmon.ko has been successfully loaded, you can start monitoring device IO either by using IO Ninja plugins (such as Serial Monitor) or by running:

$ ./tdevmon --monitor <device-name>

# for example:

$ ./tdevmon --monitor /dev/ttyUSB0

By default, all IOCTL arguments will be displayed just as numbers – the POSIX standard doesn’t specify the size of data pointed to by an IOCTL pointer argument, so tdevmon can’t uniformly extract and show the IOCTL argument data in a way which would work for all devices.

If you do need to see the contents of IOCTL pointer arguments for a particular device (e.g. termios struct when monitoring serial ports), please use the --ioctl <code>:<size> switch, e.g.:

$ ./tdevmon --monitor /dev/ttyUSB0 --ioctl 0x5402:60

This will display a hex dump of the termios struct passed as the argument for TCSETS IOCTLs.

Unloading tdevmon.ko

When you start monitoring device IO, tdevmon.ko installs hooks on the specified devices. There is a fundamental problem with any hooks, and that is the so-called hook removal problem. You can only safely remove the hook which was installed the latest – in other words, in the LIFO order.

Imagine some other module is using the same technique and it has re-hooked the device after tdevmon.ko has done so. This new module is now holding pointers to the code in tdevmon.ko. It means, that we can’t simply restore the original pointers and unload tdevmon.ko anymore – this new module can call us any time! Moreover, during its removal, this new module will restore the pointers inside the hooked file_operation struct to point back to tdevmon.ko – thinking it’s the original!

Long story short, removing hooks is much harder than installing them. The approach to the hook removal problem used in Tibbo Device Monitor is this. Unloading is split into the two stages. First, you need to stop the tdevmon.ko. During this stage, tdevmon.ko will stop dispatching notifications, stop accepting any new connections from applications willing to monitor device IO, and, most importantly, try to unhook all the devices. Naturally, this operation can fail – if tdevmon.ko detects a hook override by some other module. Only when – and if! – stop stage succeeded, you can safely unload tdevmon.ko:

$ ./tdevmon --stop
$ sudo rmmod tdevmon.ko

Note that after a successful stop tdevmon.ko no longer serves any further requests – they will fail with EBADFD error (i.e. invalid state).

It’s also possible to try and unhook a particular device without stopping tdevmon.ko:

./tdevmon --unhook /dev/ttyUSB0

When all the hooks are removed, it’s safe to issue an rmmod. Note that if there is at least one hook left, rmmod will fail with:

$ sudo rmmod tdevmon.ko
rmmod: ERROR: Module tdevmon is in use