Got a "libkmod: ERROR" log, if there is no "depmod -a" in one of initial script.

[stanepc1]

Description
Got a "libkmod: ERROR" log, if there is no "depmod -a" in one of initial script.

Steps to reproduce the issue:

1. setup an environment to test warm-boot 1000 times.
2. check log after every bootup.

Describe the results you received:
You can see "libkmod: ERROR" log sometimes.

Describe the results you expected:
There is no any "libkmod: ERROR" log for each warm-boot process.

Additional information you deem important (e.g. issue happens only occasionally):
We saw the same issue in the branch of 201803 and 201807, but if we add "depmod -a" in one of initial script in the branch of 201803 or 201807, "libkmod: ERROR" log will never appear during 1000 x 5 times warm-boot test.
Output of show version:

```

admin@sonic:~$ show version
SONiC Software Version: SONiC.master.0-406c95f
Distribution: Debian 9.5
Kernel: 4.9.0-7-amd64
Build commit: 406c95f
Build date: Thu Sep 6 06:02:06 UTC 2018
Built by: stane.shieh@VM-SONiC-stretch

Docker images:
REPOSITORY TAG IMAGE ID SIZE
docker-syncd-brcm latest ed9c3e95191c 336.4 MB
docker-syncd-brcm master.0-406c95f ed9c3e95191c 336.4 MB
docker-orchagent-brcm latest 3ce8f20611e0 257.6 MB
docker-orchagent-brcm master.0-406c95f 3ce8f20611e0 257.6 MB
docker-lldp-sv2 latest f074f651ae8d 270.9 MB
docker-lldp-sv2 master.0-406c95f f074f651ae8d 270.9 MB
docker-dhcp-relay latest 365736546300 253.7 MB
docker-dhcp-relay master.0-406c95f 365736546300 253.7 MB
docker-database latest 65f9fb7c36da 252.3 MB
docker-database master.0-406c95f 65f9fb7c36da 252.3 MB
docker-snmp-sv2 latest 742b3237ab97 291.2 MB
docker-snmp-sv2 master.0-406c95f 742b3237ab97 291.2 MB
docker-teamd latest f213143caf67 257.3 MB
docker-teamd master.0-406c95f f213143caf67 257.3 MB
docker-router-advertiser latest 0adcba677dd3 250 MB
docker-router-advertiser master.0-406c95f 0adcba677dd3 250 MB
docker-platform-monitor latest c367982a8ba8 283.3 MB
docker-platform-monitor master.0-406c95f c367982a8ba8 283.3 MB
docker-fpm-quagga latest 936906eaf468 264.1 MB
docker-fpm-quagga master.0-406c95f 936906eaf468 264.1 MB

admin@sonic:~$
```

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

[lguohan]

in your message, you said "We saw the same issue in the branch of 201803 and 201807", but the version log is kernel 4.9 which is master? which branch are you target this problem for?

[lguohan]

can you also post the full error log?

[stanepc1]

According to the information of "show version" I post, this log is in master (kernel 4.9).
This is not the first time we see it, we also saw it in the branch of 201803 and 201807.

In the branch of 201803 and 201807, we solved this issue by adding a "depmod -a " in one of initial script.

In the PR conversation of master (kernel 4.9), you request us to remove "depmod -a" in our initial script.
We perform 1000 times warm-boot test for the master (kernel 4.9), this log does appear again.
If we add "depmod -a " in our initial script for master (kernel 4.9) and re-test 1000 times warm-boot, this log will not appear.

Summary of our 1000 times warm-boot test:
If there is no "depmod -a" in one of initial script for 201803, 201807 or master, we will see following error log sometimes.

libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/evdev/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/kvm/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/sg/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/button/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/coretemp/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/crc32_pclmul/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/crct10dif_pclmul/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/ghash_clmulni_intel/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/iTCO_vendor_support/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/iTCO_wdt/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/kvm/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/linux_bcm_knet/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/linux_kernel_bde/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/nf_conntrack/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/sg/holders': No such file or directory
libkmod: ERROR ../libkmod/libkmod-module.c:1929 kmod_module_get_holders: could not open '/sys/module/shpchp/holders': No such file or directory

[lguohan]

can you share the full log? need to understand where did you get the error in the boot process.

[stanepc1]

the full log is quite large, please check it out.
full_log_libkmod.tar.gz

[lguohan]

can you get me the syslog.

in the console log, you extract related info, can you get us the full syslog?

"sudo cat /var/log/syslog.1 | grep -i error | grep -v -e "Error Record Serialization Table (ERST) support is initialized" -e "Running modprobe bridge br_netfilter failed with message: modprobe: WARNING: Module br_netfilter not found." -e "exec: \\"auplink\\": executable file not found in $PATH" -e "decode-syseeprom : ERROR : Platform did not indicate serial number" -e "Error processsing vlan attributes failed with error" -e "supervisord: start.sh isc-dhcp-relay: ERROR (no such group)" -e "o error on routing socket No buffer space available - disab" -e "usb 1-1: string descriptor 0 read error" -e "open memory.oom_control: no such file or directory""

[stanepc1]

syslog.log

[lguohan]

mostly likely it is due to the way you load those module, in your script, you are using insmod. better to use modprobe.

check the difference here.

https://askubuntu.com/questions/20070/whats-the-difference-between-insmod-and-modprobe

[stanepc1]

We use modprobe instead of insmod in our script, but the result is the same as the previous one.

We see that you merged Inventec's source code in #2010 which allows "depmod -a" in each init script.
Can you also allow "depmod -a" to come back to the PR of #2024 ?

[yxieca]

@stanepc1 sorry for leaving this issue open for so long. Can you give a problem statement as of today? If the issue has been addressed, do you mind closing this issue?

[yxieca]

Closing stale issues. Please feel free to reopen if the issue still persists.