nDPI proof of concept app runs under Windows, crashes under Linux

[mmaypo]

Hi All,
I'm looking for help on getting a simple ndpi proof of concept app going under linux (ubuntu 20, x64). I have an existing packet processing application which has a plugin architecture. The app runs under both Windows and Linux. Under linux, the plugin is a shared .so library, and under Windows, a .dll. I have compiled ndpi as a platform specific static library, which i link into the .so or .dll depending on platform. This simple test, described below, appears to be working correctly under Windows. Under linux however, it seems to crash in random locations inside a call to ndpi_detection_process_packet(). The basic use of ndpi is:

on the main plugin thread, I simply initialize ndpi:

	ndpi_ = ndpi_init_detection_module( ndpi_no_prefs );
	if (ndpi_) { 
		NDPI_PROTOCOL_BITMASK protos;
		NDPI_BITMASK_SET_ALL(protos);
		ndpi_set_protocol_detection_bitmask2(ndpi_, &protos);
		ndpi_finalize_initialization(ndpi_);
		
		...
	} 

mutliple worker threads process packets independently; each thread has it's own flowmap and holds an instance of ndpi_detection_module_struct. Each entry in a thread's flowmap holds an instance of ndpi_flow_struct named tflow.
As each packet is processed, we hand it to ndpi for processing:

		auto detected_l7_protocol = ndpi_detection_process_packet(
			ndpi_detection_module_, 
			tflow,
			layer3start,
			ip_total_len,
			packet_timestamp);

So, the same code runs on Windows, returning reasonable protocols, and crashes under Linux. GDB backtrace doesn't help too much, as the crash occurs at different locations and doesn't seem related to the line where the crash happens.
Does this sound familiar in any way? Any pointers on how to proceed at this point would be appreciated
Thanks!

[utoni]

mutliple worker threads process packets independently

How is the packet distribution done?

[mmaypo]

packets are distributed between threads by hash of 5-tuple - packets for a given flow between 2 hosts always end up on the same thread, and when there are many flows at volume (the use case for this app) traffic load is more or less balanced between threads

[utoni]

Can you share some of the backtraces?
Did you try to use memory sanitizers in your app? (and nDPI as well, see configure option --with-sanitizer; may not work on Windows except for clang)
Do you use a custom memory allocator e.g. with set_ndpi_flow_malloc()?

[mmaypo]

apologies for long delay replying... doesn't always crash in the same place, and this instance was compiled with --enable-sanitize. Not using a custom allocator.

Doesn't look too useful, but here is a stacktrace :

Thread 36 "SegmentAnalyze" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff7fbea9700 (LWP 144496)]
0x00007fffecf3a437 in ndpi_search_directconnect () from /lib/libndpi.so.4
(gdb) bt
#0 0x00007fffecf3a437 in ndpi_search_directconnect () from /lib/libndpi.so.4
#1 0x00007fffecf63ba7 in ?? () from /lib/libndpi.so.4
#2 0x00007fffecf64df0 in ndpi_detection_process_packet () from /lib/libndpi.so.4
#3 0x00007fffe5c2b200 in liveflow::FlowTracker::process_packet (this=0x61900011df80, stream_infos=, pktinfo=0x6210000e8908, timestamp=, pkt_num=)
at /home/mandrews/wp/eng/mandrews/src/liveflow/plugin/src/tcpudp_packet_info.h:25
#4 0x00007fffe5e1bff0 in liveflow::LiveFlowAPI::process_packet (timestamp=1651016903467699, pktinfo=0x6210000e8908, streaminfo=0x6210000e8900, this=0x612000cc0dc0) at /usr/include/c++/9/bits/unique_ptr.h:154
...

[utoni]

Are you sure that your C++ app was compiled with -fsanitize=address? Debugging symbols are missing as well (-g).
Can you also try to use -fsanitize=thread (clang only: can not be used together with -fsanitize=address).

Those SIGSEGVs seems to be related to your implementation instead of libndpi.

[mmaypo]

the strange thing is no issue under windows... the test is very simple: in the main thread, init detection module:

  ndpi_ = ndpi_init_detection_module( ndpi_no_prefs );
  if (ndpi_) { 
	  NDPI_PROTOCOL_BITMASK protos;
	  NDPI_BITMASK_SET_ALL(protos);
	  ndpi_set_protocol_detection_bitmask2(ndpi_, &protos);
	  ndpi_finalize_initialization(ndpi_);

	  return HE_S_OK;
  } 

pass the detection module struct to each worker thread. each worker thread holds a map of flows, and each entry in the map holds a ndpi_flow_struct. Each packet is processed with:

		auto detected_l7_protocol = ndpi_detection_process_packet(
			ndpi_detection_module_, 
			ndpiflow,
			layer3start,
			layer3len,
			packet_timestamp));

is it safe to call:

std::memset(ndpi_flow_.get(), 0 , sizeof (ndpi_flow_struct) );

if i wish to later reuse an ndpi_flow_struct?

running the app under the windows memory checker shows no issues

bt with line numbers:

(gdb) bt
#0  0x00007fffe6f8e8b2 in ndpi_parse_packet_line_info (ndpi_str=0x7bb400000000, flow=0x7b5843030800) at ndpi_main.c:6245
#1  0x00007fffe6faec38 in ndpi_check_steam_http (ndpi_struct=0x7bb400000000, flow=0x7b5843030800) at protocols/steam.c:40
#2  0x00007fffe6faf674 in ndpi_search_steam (ndpi_struct=0x7bb400000000, flow=0x7b5843030800) at protocols/steam.c:289
#3  0x00007fffe6f8bc45 in check_ndpi_detection_func (ndpi_str=0x7bb400000000, flow=0x7b5843030800, ndpi_selection_packet=443, callback_buffer=0x7bb000028000, callback_buffer_size=115) at ndpi_main.c:5016
#4  0x00007fffe6f8bfa8 in check_ndpi_tcp_flow_func (ndpi_str=0x7bb400000000, flow=0x7b5843030800, ndpi_selection_packet=0x7ffeb55be6c8) at ndpi_main.c:5087
#5  0x00007fffe6f8c027 in ndpi_check_flow_func (ndpi_str=0x7bb400000000, flow=0x7b5843030800, ndpi_selection_packet=0x7ffeb55be6c8) at ndpi_main.c:5106
#6  0x00007fffe6f8dd29 in ndpi_detection_process_packet (ndpi_str=0x7bb400000000, flow=0x7b5843030800, packet_data=0x7ffec567c278 "E", packetlen=1500, current_time_ms=1651092461122) at ndpi_main.c:5927
#7  0x00007fffe6e7260d in liveflow::FlowTracker::process_packet (this=0x7b64dc04b000, stream_infos=0x7b4400a72ce0, pktinfo=0x7b8000002b08, timestamp=1651092461122027, pkt_num=15)
    at /home/mandrews/wp/eng/mandrews/src/liveflow/plugin/src/flow_tracker.cpp:895

[utoni]

pass the detection module struct to each worker thread. each worker thread holds a map of flows, and each entry in the map holds a ndpi_flow_struct. Each packet is processed with:

Unfortunately, every thread need's it's own struct ndpi_detection_module_struct. This is a limitation of nDPI as it does not support any thread synchronization for several reasons.

FYI: #1344

[mmaypo]

well that would be the issue -my bad