Access Denied (winerror 5) in Serial/Pipe/Mailslot Monitors on Windows

Symptoms

Starting the Serial Monitor, Named Pipe Monitor or Mailslot Monitor plugin and pressing the Capture button yields:

Session started
Cannot start capture: Access is denied.

Details

IO Ninja uses a kernel mode filter driver (a part of Tibbo Device Monitor package) to intercept communications between applications and drivers. Obviously, it imposes a certain security risk, so by default we only allow Administrators to access this facility.

You can check the current security descriptor by opening a Windows Command Prompt, navigating to C:\Program Files\Tibbo\DeviceMon 3\bin and then executing:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --show-sd

You may see:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --show-sd
OWNER: SYSTEM S-1-5-18
GROUP: SYSTEM S-1-5-18
ALLOW: Administrators S-1-5-32-544

… or, which is functionally equivalent:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --show-sd
No valid security descriptor set (defaults will be used by tdevmonc.sys)

If Windows UAC (User Account Control) is OFF, then being in the Administrators group is enough – Serial Monitor should cause no problems. However, if UAC is ON, then you may receive Access Denied error whether or not you are in the Administrators group.

Solution

To combat this, you can either:

  • Disable UAC

… or:

  • Run IO Ninja as Administrator

… or:

  • Add yourself to the allowed list

The first and second options are self-explanatory but rather inconvenient. Therefore, the recommended solution is to add yourself to the allowed list.

To do so, open an elevated Command Prompt (find “Command Prompt” in the Start menu, right click on it and select “Run as Administrator”). Then type:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --allow Vladimir
OWNER: SYSTEM S-1-5-18
GROUP: SYSTEM S-1-5-18
ALLOW: Administrators S-1-5-32-544
ALLOW: Vladimir S-1-5-21-1208373166-1502685412-2756468959-1000

Obviously, instead of “Vladimir” you should type your own username. You can repeat the same command to add more users or user groups to the allowed list. Sometimes it may be necessary to deny (rather than allow) certain users or groups. Do so with:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --deny Guest
OWNER: SYSTEM S-1-5-18
GROUP: SYSTEM S-1-5-18
DENY:  Guest S-1-5-21-1208373166-1502685412-2756468959-501
ALLOW: Vladimir S-1-5-21-1208373166-1502685412-2756468959-1000

Note that if a user is both on ALLOW and DENY list, he will not be allowed to monitor (DENY entries take precedence).

To get a full list of usernames and groups available on your machine for your reference, type:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --list-all-accounts

If you want to reset everything to defaults, type:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --set-default-sd
OWNER: SYSTEM S-1-5-18
GROUP: SYSTEM S-1-5-18
ALLOW: Administrators S-1-5-32-544

Clearing the security descriptor (nobody will be allowed to monitor) can be done with:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --clear-sd
OWNER: SYSTEM S-1-5-18
GROUP: SYSTEM S-1-5-18
DACL is empty

After you finished configuring the Security Descriptor, make sure you are really allowed to monitor without elevation. To do so, start a regular (non-elevated) command prompt, and type:

"C:\Program Files\Tibbo\DeviceMon 3\bin\tdevmon.exe" --check-access
Access granted

Now when you start IO Ninja and run Serial Monitor you should not see the Access Denied error.