Can the Ethernet Tap also work with Wireshark?

Hello!
I've been trying an Ethernet tap. I've been working with other ones (eg the Dualcomm one) but I prefer the ioninja approach, because is less messy connecting it, as it uses a "simple" USB cable instead of a Ethernet + power cable in the other cases.
But I found some serious troubles. For me it is vital to be able to use it to feed directly Wireshark. Let me report you:

• it seems that ioninja-hwc does not timestamp the frames. E.g. I tried "ioninja-hwc --ethernet-tap --pcap | wireshark -k -i -". It works, but the timestamp is weird
  
  Doing something similar with, for example, tcpdump (tcpdump -i any -s0 -U -w - | wireshark -k -i -), gives a correct timestamp
  

• the former ioninja-hwc command usually "stops" capture packets a few seconds after start. Again I compared that with what happens using the "tcpdump", and using "ioninja-hwc" it usually happens after capturing a few hundreds of frames, whilst with "tcpdump" there is no problem after capturing hundreds of thousands.
  
  In some way, ioninja-hwc (+wireshark) seems to work better if firstly a .pcap file is generated, and then open (ioninja-hwc --ethernet-tap --pcap > tmp.pcap) . But this is quite inconvenient, as what I want is "live capture".
  As I commented, the usual is found this problem, but it seems that in some cases it works well for more time or indefinitely (?). Using named pipes (mkfifo) does not change the behavior. I did this test in my dear Raspberry+Arch system. Then, I did it also in Windows, and at first I thought that it works well (as it works several minutes ok), but then I forced more traffic through the interface (not to much, simply doing a "find /" on the device which is "sniffed" (which is accessed via ssh)), and then the same problem shows up
  

I also found a problem with the "ioninja" main application in my Linux platform (not in Windows). If I tried to open a "Ethernet-tap session", a shared library error appears...


... I tried to download previous versions of libpcap, and then use "LD_PATH_LIBRARY" acordingly, but I was unable to workaround the issue.
Anyway, for me, in first instance, it would be more important to solve the problems related to "live capture" with wireshark

Regards!
Josep

Hello Jose,

The issues are confirmed and (mostly) fixed already. I will try to make a service release this week. I can also make an RPi build for you to try first.

Hello Vladimir,
Thank you! From my side, I am not in a hurry, so take your time in order to do it in the most convenient way for you.
Also, for the record, I wrote the post very quickly, and I forgot to mention another advantages I find to your tap. Besides less cabling and no "external" power supply required, it is also nice do not use an ethernet interface in the host device (it is easier to find USB ports, and RJ45 are increasingly hard to find in PC), and also -specially if you end up with an external USB to RJ45 converter attached to the PC- there are some chances that these NIC does not support promiscuous mode, so you can not use for capturing.
Regards
Josep

Hello again Jose,

Thanks for your feedback on our taps! I also believe that using the USB for sniffing instead of a NIC (even a virtual one, i.e., when plugging a tap essentially creates a dedicated NIC for sniffing) is a more convenient and overall better approach.

Regarding issues you described above -- please try the latest build for RPi:

https://tibbo.com/downloads/archive/ioninja/ioninja-5.3.2/ioninja-5.3.2-b-linux-arm32.tar.xz
https://tibbo.com/downloads/archive/ioninja/ioninja-5.3.2/ioninja-5.3.2-b-linux-arm64.tar.xz

All of those should be fixed (please do let me know if not).

(ops... I wrote a quite long post, reporting an issue, but I think I did something basically wrong... let me check...)

Hello Vladimir!
Well, unfortunatelly, I'm going to re-post more or less what I wrote yesterday, as I did try the new build, and I found that something seems not ok. I deleted the post because I realized that the link I was trying is a gigabit one (so, my mistake), but today I replied the test forcing a 100Mbit/s link, and I found a similar issue. Let me report you.
Just starting to capture with the new build, I've the "feeling" that "something is lost / wrong".
So I tried the following, I capture (tcpdump -i end0 -s0 -U -w - | TMPDIR=/root/tmp00 wireshark -k -i -) in the same interface that I put the tap (ioninja-hwc --ethernet-tap --pcap | TMPDIR=/root/tmp00 wireshark -k -i -). The pcap should be the same. Those two commands are issued in the same device.
I've captured data for a minute -simultaneously-, and I filtered both pcaps preciselly.

Also I have done "pings" to the device at the beginning and end of the capturing, in order to have some reference.
The load is a bit large (there are a couple of X11 over SSH sessions) but it seems perfectly realistic case (the interface seems to work a 10 Mbit/s).
Doing this, I saw that the "tcpdump" capture is 744k packets, whilst the ioninja-hwc is 133k. So many packets are lost


Also, if I look for the pings, I can see that in the "tcpdump" there are all that I generated (from 172.31.25.2) and the replies, whilst with "ioninja-hwc" there are few pings. This is also useful to see that both pcaps are in sync.


Also I attach a IO Graph for both cases:


I have also checked the USB that it is used to connect the tap, and I is working as "Hi-Speed" (480 Mbit/s).

And (today), I also check the link speed

And the ethernet conversation, in which -again- it seems that many errors are injected in the ioninja-hwc pcap, (the tcpdump one is perfectly normal)

Well, this is a quite issue, as it seems not be reliable, and just -when you are using a tap- is because you are looking for some kind of problem on the "device under test", and you do not want to worry about the "tap" device.

If you need any more data, please let me know. I hope I did (again) something wrong and it is my fault (o -if I am right- that it would be easy to fix).

Regards!
Josep

Hello Jose,

Thank you so much for a great report!

It took me some time to check and test everything; please read my comments below.

You expect pcap dumps captured with ioninja-hwc and with tcpdump to be exactly the same. Actually, that's not guaranteed. Ethernet Tap does not "pause" forwarding of Ethernet frames from one port to the other if the Cypress USB buffer is full -- that's in line with how other taps for IO Ninja work (they don't even "forward" data and are simply connected to the existing lines instead). The buffer on the Ethernet Tap itself is very small (as it has no RAM/flash), so if the software is not fast enough to read the captured data -- data will be lost).

The packet loss ratio you experienced with ioninja-hwc is, of course, not acceptable. The original version of ioninja-hwc was doing the initiating of USB read transfers, parsing the input from the Tap and writing it to the output file (a pipe to Wireshark, in your case) all in one thread. So if somewhere along the way any delay happens, then new USB reads are not scheduled, the Cypress USB buffer on the Ethernet Tap gets full, and the packets are lost.

I rewrote ioninja-hwc so that scheduling of USB reads is now done in a dedicated thread, so any blocking waits during writing to the output file/pipe should not prevent new USB reads from being scheduled. Also, it's now directly controllable how many simultaneous USB read transfers are passed to the bus driver (as well as buffer sizes for those).

Please try this build:

https://tibbo.com/downloads/archive/ioninja/.internal/ioninja-hwc-test.tar.xz

Controlling the buffer sizes and read parellelism is done via --read-parallelism and --read-block-size command line parameters. It also generates a log of overflows in %HOME%/ioninja-hwc-log.txt

Still, this can't completely prevent packet loss -- Linux/Windows/macOS are not realtime OSes, and at 10+ MBytes per second, any delay could result in an overflow of the (rather tiny) buffer on the Ethernet Tap. Considering this, it's recommended to connect Ethernet Tap to a fast host to minimize packet loss.

The ultimate solution should include some RAM/SD/eMMC on the Ethernet Tap to store captured packets.

Besides packet loss, there's a bigger problem.

Under some conditions during heavy load, the FPGA firmware could send garbaged data over USB (not only the packet contents, but the USB protocol headers could suffer as well). Luckily -- or, maybe, unfortunately (because it essentially hid the problem) -- our USB protocol is designed in a way so that it recovers and re-synchronizes almost instantly, so this problem remained unnoticed. But yes, I was able to reproduce the issue. Data corruption over a bulk USB endpoint is not acceptable and should never happen. Our FPGA developer is investigating the issue at the moment. Hopefully, we will have a firmware patch for this soon.

Once again, thank you for reporting the issue in great detail; I'll keep you posted.

Hello Vladimir,
finally I've a few minutes to try the build. There is a significant improvement!, but under the same type of test, there are still losses.

This corresponds to 72 s capture (tcpdump on the interface vs hwc), and the displayed frames are ICMP ones (16 is the right value).
And, logically, it remains the issue of the MAC address.
I'd like to perform the same test, but under a slightly lower load, and also try using a more powerful platform as host for the ioninja-hwc, but I did not have time. Sorry. If (when) I eventually could, I will report you.
Regards!
Josep

Hello Josep,

Wow, it took us quite some time to tackle this issue, but the newly released ioninja-5.4.0 finally offers a fix. Both ioninja-hwc and the FPGA firmware/config had undergone serious modifications to eliminate the packet loss you experienced under (relatively) high load.

Please try the new build and let me know how it works for you -- but make sure the FPGA firmware is up to date before your tests. The easiest way to do so is to plug the Ethernet Tap into your laptop and start an Ethernet Tap session in IO Ninja -- it will upload the new FPGA firmware on the first run.

If you have to do it from the command line, then it's:

$ ./ioninja-hwc --ethernet-tap --firmware=./scripts/plugins/EthernetTap/firmware/eth.rpd

Great news!! Thank you, I try it and tell you ASAP.

Hello Vladimir,
I tried it, but I found a problem. I tried to do tha same, but an error appears:

I think it could simple be a issue with the generation of the pcap file by the ioninja-hwr.

If instead of pipe the output, I save it to a file, and then open it with wireshark, the same error appears.

As complementary piece of info, the ethernet tap FW upgrade from the IONINJA works like a charm
... but in first place I tried the command line option and does not work: I issued the command and nothing seems to happen... after many time (> 30 min) I decided CTRL-C it (shiver down my spine). It terminates well, and then I tried the IONINJA option that works well, so nothing bad seems to have happend.

Let me know if I can provide more info.

Regards!
Josep

Just checked. You are right, ioninja-hwc writes some extra notifications to the stdout; as such, it can't be piped to wireshark normally, i.e., the way you tried. This will be fixed in the upcoming service release, of course.

Meanwhile, you can use a mkfifo to create a named pipe, then run ioninja-hwc as such:

./ioninja-hwc --ethernet-tap --pcap --out=your-pipe-name

Re the --firmware switch -- I got confused, actually. It doesn't immediately upload an FPGA firmware and instead is utilized in our test fixture. But yeah, it currently doesn't really work as people would expect. So I think I'll modify that (i.e., make it upload an FPGA firmware from the command line).

Hello Vladimir!

Using named pipes works perfectly. I quickly perform the same test I did, in three platforms: a Rasperry PI4, a Rock64 and a quite old PC (Pentium T4300 @ 2.10GHz) with Linux.

The results seems really good! Hats off!! because the task did not seem easy.

I will put some of my typical screens below, but basically: there are no estrange MAC in the conversations screen; all ICMP are in place; the captured bytes are aligned; the average MByte/s are about 7 (Rock64 cap), 25 (R PI4 cap) and 32 (x86-64 cap) and in all cases the results are coherent... Overall: "the feeling" is that all is ok

As you point out in one of your first emails, it seems not possible to have a identical pcap from tcpdump and a tap (eg. I see that the tap capture has Ethernet Frame checks, and these tcpdump ones do not (so the byte counter are different), I have doubts about how the NIC/kernel/tcpdump works, and also how frames are managed at the TAP and how are sent to the host; it seems that there are jumbo (and superjumbo.) frames (in the x84, in the tcpdump, it seems that there are many really big jumbograms); etc... (please forgive me if I said some incorrect, because this is a bit above my expertise... feel free to correct me)

Well, I put below some of the captured data fyi. Thank you, and congratulations for your work!
Regards
Josep

Rock64 - TCPDUMP

Rock64 - Tap

Raspberry PI 4 - TCPDUMP

Raspberry PI 4 - Tap