Installation of Tibbo Device Monitor on Linux

Preface

Tibbo Device Monitor service provides device monitoring capabilities for IO Ninja. It’s the engine behind the Serial Monitor plugin, Named Pipe Monitor plugin and such. Monitoring of device IO on Linux is currently performed by hooking the file_operations structure of the corresponding device, but the hooking details and techniques may change in the future.

Installation

Download the latest tdevmon-linux-<version>-<arch>.tar.xz tarball from:

http://ioninja.com/downloads

Unpacking it yields the following directory structure:

bin/
include/
    tdevmon/
lib/
    cmake/
        tdevmon/
share/
    tdevmon/
        license/
        src/
            lkm/

The core of Tibbo Device Monitor service on Linux consists of the following two main components:

  • Service management application tdevmon
  • Loadable kernel module tdevmon.ko

The tdevmon executable resides in bin/ directory and can be executed directly. Run ./tdevmon --help for available options.

The loadable kernel module (LKM) tdevmon.ko cannot be distributed as a pre-compiled binary (well, technically speaking, it can, but that would be a wrong thing to do). Instead, it must be built against the exact version of Linux kernel running on your machine. The sources of tdevmon.ko can be found in share/tdevmon/src/lkm.

Building tdevmon.ko

First of all, you need to install the system headers required to build Linux kernel modules. The actual way of doing so depends on the Linux distribution; usually, the package with the required headers is called linux-headers, but it may vary.

On Ubuntu and other Debian-based distributions, run:

$sudo apt-get install linux-headers-$(uname -r)

On Arch Linux, run:

$sudo pacman -S linux-headers

… and so on. Refer to your distribution reference on building Linux kernel modules.

After that, navigate to the share/tdevmon/src/lkm folder and inside it simply run make:

$cd <tdevmon-dir>/share/tdevmon/src/lkm
$make
make -C /lib/modules/4.12.3-1-ARCH/build/ M=<tdevmon-dir>/share/devmon/src/lkm modules
make[1]: Entering directory '/usr/lib/modules/4.12.3-1-ARCH/build'
  CC [M]  <tdevmon-dir>/share/devmon/src/lkm/module.o
  CC [M]  <tdevmon-dir>/share/devmon/src/lkm/Device.o
  CC [M]  <tdevmon-dir>/share/devmon/src/lkm/Hook.o
  CC [M]  <tdevmon-dir>/share/devmon/src/lkm/Connection.o
  CC [M]  <tdevmon-dir>/share/devmon/src/lkm/HashTable.o
  CC [M]  <tdevmon-dir>/share/devmon/src/lkm/ScatterGather.o
  CC [M]  <tdevmon-dir>/share/devmon/src/lkm/lkmUtils.o
  LD [M]  <tdevmon-dir>/share/devmon/src/lkm/tdevmon.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      <tdevmon-dir>/share/devmon/src/lkm/tdevmon.mod.o
  LD [M]  <tdevmon-dir>/share/devmon/src/lkm/tdevmon.ko
make[1]: Leaving directory '/usr/lib/modules/4.12.3-1-ARCH/build'

Congratulations! tdevmon.ko is built.

Loading tdevmon.ko

To load tdevmon.ko into Linux kernel, run from within the same folder:

sudo insmod tdevmon.ko

This will create a device called /dev/tdevmon which can be used to communicate with the kernel module and install or remove hooks on selected devices.

Permissions

You have to realize, that providing device IO monitoring capabilities introduces certain security risks. Therefore, by default the /dev/tdevmon device only allows root access; non-superusers simply would not be able to open it. During debugging, however, it may be desirable to alleviate access limitations. You can do so with either changing permissions/ACLs of the /dev/tdevmon post-creation, or by overriding the permissions parameter while loading tdevmon.ko:

sudo insmod tdevmon.ko permissions=0666

The command above will allow read-write access to /dev/tdevmon for everybody.

To ensure tdevmon.ko is properly loaded and configured, please navigate to bin/ and run:

$./tdevmon --stats
tdevmon.ko stats:

Total devices:    0
Connections:      0

Now you can start monitoring device IO either by using IO Ninja plugins (such as Serial Monitor) or by running:

$./tdevmon --monitor <device-name>

# for example:

$./tdevmon --monitor /dev/ttyUSB0

Unloading tdevmon.ko

When you start monitoring device IO, tdevmon.ko installs hooks on the specified devices. There is a fundamental problem with any hooks, and that is the so-called hook removal problem. You can only safely remove the hook which was installed the latest – in other words, in the LIFO order.

Imagine some other module is using the same technique and it has re-hooked the device after tdevmon.ko has done so. This new module is now holding pointers to the code in tdevmon.ko. It means, that we can’t simply restore the original pointers and unload tdevmon.ko anymore – this new module can call us any time! Moreover, during its removal, this new module will restore the pointers inside the hooked file_operation struct to point back to tdevmon.ko – thinking it’s the original!

Long story short, removing hooks is much harder than installing them. The approach to the hook removal problem used in Tibbo Device Monitor is this. Unloading is split into the two stages. First, you need to stop the tdevmon.ko. During this stage, tdevmon.ko will stop dispatching notifications, stop accepting any new connections from applications willing to monitor device IO, and, most importantly, try to unhook all the devices. Naturally, this operation can fail – if tdevmon.ko detects a hook override by some other module. Only when – and if! – stop stage succeeded, you can safely unload tdevmon.ko:

$./tdevmon --stop
$sudo rmmod tdevmon.ko

Note that after a successful stop tdevmon.ko no longer serves any further requests – they will fail with EBADFD error (i.e. invalid state).

It’s also possible to try and unhook a particular device without stopping tdevmon.ko:

$./tdevmon --unhook /dev/ttyUSB0

When all the hooks are removed, it’s safe to issue a rmmod. Note that if there is at least one hook left, rmmod will fail with:

$sudo rmmod tdevmon.ko
rmmod: ERROR: Module tdevmon is in use