Access Denied (winerror 5) in Serial/Pipe/Mailslot Monitors on Windows

Symptoms

Starting the Serial Monitor, Named Pipe Monitor or Mailslot Monitor plugin and pressing the Capture button yields:

Session started
Cannot start capture: Access is denied.

Details

This error is caused by either of the following reasons:

  1. Lack of permissions to access Tibbo Device Monitor;
  2. The target serial port is opened in another application (in case of the Serial Monitor plugin).

Tibbo Device Monitor

IO Ninja uses a kernel mode filter driver (a part of Tibbo Device Monitor package) to intercept communications between applications and drivers. Obviously, it imposes a certain security risk, so by default we only allow Administrators to access this facility.

You can check the current security descriptor by opening a Windows Command Prompt, navigating to C:\Program Files\Tibbo\DeviceMon 3\bin and then executing:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --show-sd

You may see:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --show-sd
OWNER: SYSTEM S-1-5-18
GROUP: SYSTEM S-1-5-18
ALLOW: Administrators S-1-5-32-544

… or, which is functionally equivalent:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --show-sd
No valid security descriptor set (defaults will be used by tdevmonc.sys)

If Windows UAC (User Account Control) is OFF, then being in the Administrators group is enough – Serial Monitor should cause no problems. However, if UAC is ON, then you may receive Access Denied error whether or not you are in the Administrators group.

Why Do I Need Exclusive Access To A Serial Port?

To monitor a device, Tibbo Device Monitor must attach its filtering device to the target device stack, and in order to do that, the target device must be opened. On Windows, most serial port drivers create their respective serial devices using the DO_EXCLUSIVE flag, which allows one – and only one! – open operation at any given time. In other words, most serial devices on Windows enforce exclusive access – when one application opens a port, any attempt to open the same port in another application will fail with the Access is denied error.

Note

After Tibbo Device Monitor successfully completes the attach operation, it is no longer required to open the target device to start monitoring. It means, that if you were able to start monitoring some serial device at least once during your Windows session, you now can start monitoring the same device at any time – even when this device is opened in another application.

Solution

In case of the Serial Monitor plugin, first of all, make sure you do have exclusive access to the target device.

Next, make sure you do have permissions to access Tibbo Device Monitor. To achieve that, you can either:

  1. Disable UAC (your user must be in the Administrators group);

… or:

  1. Run IO Ninja as Administrator;

… or:

  1. Start an IO Ninja monitoring plugin (e.g., Serial Monitor) as Administrator;

… or:

  1. Add yourself to the so-called allowed list of Tibbo Device Monitor.

The first three options are self-explanatory, but may sometimes be not too convenient. Therefore, the recommended solution is option 3 – add yourself to the allowed list.

To do so, open an elevated Command Prompt (find “Command Prompt” in the Start menu, right click on it and select “Run as Administrator”). Then type:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --allow Vladimir
OWNER: SYSTEM S-1-5-18
GROUP: SYSTEM S-1-5-18
ALLOW: Administrators S-1-5-32-544
ALLOW: Vladimir S-1-5-21-1208373166-1502685412-2756468959-1000

Obviously, instead of “Vladimir” you should type your own username. You can repeat the same command to add more users or user groups to the allowed list. Sometimes it may be necessary to deny (rather than allow) certain users or groups. Do so with:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --deny Guest
OWNER: SYSTEM S-1-5-18
GROUP: SYSTEM S-1-5-18
DENY:  Guest S-1-5-21-1208373166-1502685412-2756468959-501
ALLOW: Vladimir S-1-5-21-1208373166-1502685412-2756468959-1000

Note that if a user is both on ALLOW and DENY list, he will not be allowed to monitor (DENY entries take precedence).

To get a full list of usernames and groups available on your machine for your reference, type:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --list-all-accounts

If you want to reset everything to defaults, type:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --set-default-sd
OWNER: SYSTEM S-1-5-18
GROUP: SYSTEM S-1-5-18
ALLOW: Administrators S-1-5-32-544

Clearing the security descriptor (nobody will be allowed to monitor) can be done with:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --clear-sd
OWNER: SYSTEM S-1-5-18
GROUP: SYSTEM S-1-5-18
DACL is empty

After you finished configuring the Security Descriptor, make sure you are really allowed to monitor without elevation. To do so, start a regular (non-elevated) command prompt, and type:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --check-access
Access granted

Now when you start IO Ninja and run Serial Monitor you should not see the Access Denied error.