Kernel panic на ядре 4.9

[omikhaylov]

Описание

Ядро 4.9 валится при выполнении iptables -A FORWARD -m ndpi --proto tiktok -j ACCEPT

nDPI Environment (please complete the following information):

• ОС: Debian.
• Версия ОС: Stretch
• Архитектура: x86_64
• Коммит: 845399a
• Запущено на ESXi

Воспроизведение

Собранные заранее файлы были помещены в соответствующие директории:

• "/lib/modules/4.9.0-11-amd64/extra/" для "xt_ndpi.ko"
• "/usr/lib/x86_64-linux-gnu/xtables/" для "libxt_ndpi.so"
  При выполнении iptables -A FORWARD -m ndpi --proto tiktok -j ACCEPT появлялась следующая ошибка:

[  591.611881] ------------[ cut here ]------------
[  591.613108] kernel BUG at /kernel_11/linux-4.9.189/mm/vmalloc.c:1375!
[  591.614716] invalid opcode: 0000 [#1] SMP
[  591.615751] Modules linked in: iptable_filter ip6_tables xt_ndpi(O) st_gost(O) cp_aes(O) cp_plg1(O) vpnirq(O) sobol(PO) nls_ascii edac_core crct10dif_pclmul crc32_pclmul ppdev nls_cp437 vmw_balloon ghash_clmulni_intel intel_rapl_perf vfat fat joydev serio_raw evdev vmwgfx ttm drm_kms_helper drm vmw_vmci sg shpchp parport_pc parport ac button autofs4 ts_bm vpndrvr(O) cryptopm(O) ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache ebtable_filter ebtable_nat ebtables iptable_nat ip_tables nf_nat_ftp nf_nat_ipv4 nf_nat nf_conntrack_ftp nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack x_tables sd_mod ata_generic crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse ata_piix pcnet32 mii mptspi scsi_transport_spi mptscsih mptbase i2c_piix4 libata scsi_mod floppy
[  591.635363] CPU: 0 PID: 1331 Comm: iptables Tainted: P           O    4.9.0-11-amd64 #1 Debian 4.9.189-3st1
[  591.637668] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[  591.640206] task: ffff8cffb2aa5000 task.stack: ffffb9b640dbc000
[  591.641633] RIP: 0010:[<ffffffff857cb6d2>]  [<ffffffff857cb6d2>] __get_vm_area_node+0x132/0x140
[  591.643800] RSP: 0018:ffffb9b640dbf950  EFLAGS: 00010206
[  591.645118] RAX: 0000000080000200 RBX: 000000000000bbe0 RCX: ffffb9b640000000
[  591.646908] RDX: 0000000000000022 RSI: 0000000000000001 RDI: 000000000000c000
[  591.648669] RBP: 000000000000bbe0 R08: ffffd9b63fffffff R09: 00000000ffffffff
[  591.650508] R10: 0000000000000001 R11: 0000000000100000 R12: 0000000000000090
[  591.652259] R13: 00000000024000c2 R14: ffff8cffb4eea000 R15: ffffffffc05f620a
[  591.654029] FS:  00007f64ad4a0280(0000) GS:ffff8cffbd600000(0000) knlGS:0000000000000000
[  591.656016] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  591.657498] CR2: 000055df41343128 CR3: 0000000037a1c000 CR4: 0000000000160670
[  591.659311] Stack:
[  591.659908]  ffffffff857ccea0 00000001024000c2 ffffffffc059ee2a ffff8cffb4eea038
[  591.661945]  000000000000000d ffff8cffb4ed29c0 8000000000000163 000000000000bbe0
[  591.663867]  000000000000012c 0000000000000090 0000000000000000 ffff8cffb4eea000
[  591.665842] Call Trace:
[  591.666458]  [<ffffffff857ccea0>] ? __vmalloc_node_range+0x70/0x280
[  591.668009]  [<ffffffffc059ee2a>] ? ndpi_malloc+0x1a/0x30 [xt_ndpi]
[  591.669508]  [<ffffffff857cd391>] ? vmalloc+0x51/0x60
[  591.670731]  [<ffffffffc059ee2a>] ? ndpi_malloc+0x1a/0x30 [xt_ndpi]
[  591.673213]  [<ffffffffc059ee2a>] ? ndpi_malloc+0x1a/0x30 [xt_ndpi]
[  591.675583]  [<ffffffffc059ef6b>] ? ndpi_calloc+0x1b/0x40 [xt_ndpi]
[  591.677959]  [<ffffffffc05afbec>] ? ndpi_set_protocol_detection_bitmask2+0xd78c/0xe220 [xt_ndpi]
[  591.680861]  [<ffffffffc058826d>] ? ndpi_enable_protocols+0x9d/0xb0 [xt_ndpi]
[  591.683403]  [<ffffffffc0588730>] ? ndpi_mt_check+0x1b0/0x210 [xt_ndpi]
[  591.685800]  [<ffffffff85954379>] ? list_del+0x9/0x30
[  591.687790]  [<ffffffff85954379>] ? list_del+0x9/0x30
[  591.689780]  [<ffffffff857e692a>] ? page_is_poisoned+0xa/0x20
[  591.692189]  [<ffffffff8578c6d0>] ? get_page_from_freelist+0x8f0/0xb20
[  591.694507]  [<ffffffff857aa470>] ? pcpu_alloc_area+0x220/0x410
[  591.696651]  [<ffffffffc01bbe6d>] ? xt_check_match+0x9d/0x1c0 [x_tables]
[  591.698924]  [<ffffffff857aa23b>] ? pcpu_next_unpop+0x3b/0x50
[  591.701015]  [<ffffffff857ab247>] ? pcpu_alloc+0x3a7/0x680
[  591.703016]  [<ffffffffc01ba950>] ? xt_find_match+0xf0/0x100 [x_tables]
[  591.705250]  [<ffffffffc0216276>] ? find_check_entry.isra.8+0x116/0x280 [ip_tables]
[  591.707721]  [<ffffffffc02171af>] ? translate_table+0x43f/0x5d0 [ip_tables]
[  591.710059]  [<ffffffffc0217e58>] ? do_ipt_set_ctl+0x118/0x1c5 [ip_tables]
[  591.712349]  [<ffffffff85b4d0b4>] ? nf_setsockopt+0x44/0x70
[  591.714407]  [<ffffffff85af4b02>] ? SyS_setsockopt+0x82/0xf0
[  591.716446]  [<ffffffff85603b7d>] ? do_syscall_64+0x8d/0x100
[  591.718462]  [<ffffffff85c1c44e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
[  591.720780] Code: 00 83 f9 1e 0f 4f c8 49 d3 e2 4d 89 d4 e9 38 ff ff ff 4c 89 ff e8 8f fd 01 00 48 83 c4 08 31 c0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 <0f> 0b 66 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 83
[  591.730043] RIP  [<ffffffff857cb6d2>] __get_vm_area_node+0x132/0x140
[  591.732356]  RSP <ffffb9b640dbf950>
[  591.733941] ---[ end trace 7c88c0909b62266b ]---
[  591.735691] Kernel panic - not syncing: Fatal exception in interrupt
[  591.738036] Kernel Offset: 0x4600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  591.742158] Rebooting in 300 seconds..]

Дополнительный контекст

Сборка проводилась в Docker. Использовалcя пакет linux-headers.
Сборка из коммита 3ff5a9b работала исправно.

[vel21ripn]

В 4.9.189 mm/vmalloc.c:1375 BUG_ON(in_interrupt());
IMHO xt_check_match никогда не выполняется во время прерывания.
Если сделать исправление

--- main.c	2023-02-17 12:35:59.247027489 +0300
+++ main.c.vmalloc	2023-03-13 23:52:52.020712420 +0300
@@ -595,7 +595,7 @@
 
 static void *malloc_wrapper(size_t size)
 {
-	if(size > 32*1024) {
+	if(0 && size > 32*1024) {
 		/*
 		 * Workarround for 32bit systems
 		 * Large memory areas (more than 128 KB) are requested

баг проявится?

[omikhaylov]

После исправления баг не проявился.

[vel21ripn]

В ближайшее время сделаю более корректное изменение.

[BrainSlayer]

you need to refine your code. the check for 32kb is wrong. often kernels are patched with modified memory configurations and it might also be architecture dependend and allocator depenend (SLAB, SLUB etc). in addition you check for 32 kb not 32mb which is very small and unneccessary.
kmalloc can safly allocate up to KMALLOC_MAX_SIZE (default is 32mb on slab. and 4mb for slub). but your malloc wrapper may also return NULL in case a large chunk is allocated using GFP_ATOMIC since this is a time critical allocation.

so my advise. check for KMALLOC_MAX_SIZE. so standard kernel kmalloc will be used in most cases. and maybe think about handling failed allocations for gfp_atomic if there is such a case

[vel21ripn]

@BrainSlayer Thanks for the hint about KMALLOC_MAX_SIZE.
This code was a temporary workaround for a memory allocation problem in the kernel for some non-X86 architecture. There was a problem to allocate 128kB of RAM.
For 2019, during the operation of the module, it was not required to allocate more than 32 kB of memory. The only place where it was necessary to allocate more than 32 kB of memory was the ndpi_init_detection_module() call.

[BrainSlayer]

@vel21ripn yes this might happen on embedded systems with modified kernels which restrict the kmalloc maximum size (which depends on the page size and custom kernel konfiguration). i had the same experience while developing the zstd implementation for zfs in the past. just take care that in case of GFP_ATOMIC nothing is guaranteed. no matter which size. so null checks are very important here. its all about timing. GFP_ATOMIC is restricted to a very short allocation time. if it fails, it returns NULL. also on very small block sizes. GFP_KERNEL however will always return a valid unfragmented pointer of memory is available up to KMALLOC_MAX_SIZE. everything bigger immediatly returns NULL and needs vmalloc fallback then.

[BrainSlayer]

also a hint. i made some optimizations for your current ndpi flow4 tree in dd-wrt. it does compile everything as whole source which all functions declared as static. so like lto for poors in kernel space. this may increase performance and definitly decreases codesize. you may merge my approach, so i dont have to spend too much time in syncing sources over and over again if there are any updates here.

[vel21ripn]

@BrainSlayer Where can I see your build for dd-wrt?
The biggest problem with the main project is the static ndpi_packet_struct which precludes the use of multithread.
Almost every "git merge" leads to conflicts and their resolution takes time.

[BrainSlayer]

the builds can be downloaded from our website. the source can be found in our svn or at https://github.com/mirror/dd-wrt/tree/master/src/router/ndpi-netfilter

its in sync with your sources almost but contains alot of small changes for compiling all static. nothing really serious. just compare and review it to understand how its made.
but i understand if its hard for merging with upstream. i have the same issue everytime i sync your sources with my tree. i do this all manually and it takes some hours every time to get all to work

[omikhaylov]

В ближайшее время сделаю более корректное изменение.

Были ли в недавних коммитах исправления?

[vel21ripn]

Да. Если все будет нормально, то завтра выложу.
Там и обновление и исправление ошибок. ntop#1948 IMHO полезный.

[vel21ripn]

Существенный коммит f64efcf