Version 0.22.0 Docker image broken, "insufficient memory for the Java Runtime Environment"

[erkexzcx]

Version 0.21.0 is working fine, but it's broken on 0.22.0.

This is what I am getting when opening docker logs -f akhq

[root@dockerhost _data]# docker logs -f akhq
[0.004s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /app/hs_err_pid8.log
[0.004s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.

Other notes that might help:

• I am using default JVM OPTS.
• Docker image: https://hub.docker.com/r/tchiotludo/akhq
• 8 Kafka clusters (6 servers per cluster), no authorization (only bootstrap.servers field is specified).

[jheinitz]

Hello!

We are facing the same issue here. We have three clusters and two of them work fine after upgrading to 0.22.0, but the third one is throwing exactly the same error message. The only difference I can see is that the one where it is not working is a RedHat 8.5 VM while the other two are on RedHat 8.6.

The strange thing is that the VM where it is not working has 32GB RAM while the other two only have 16GB RAM. And there is more free RAM available on the machine with 32GB where it is not working.

Any idea why this happens?

Kind regards

Jens

[tchiotludo]

The container start and the ui is reachable or don't start at all ?

[jheinitz]

To me it looks like it doesn't even start the JVM. This is the difference between a working instance and a failing instance:

Failing:

root@akhq-abn app]# docker run tchiotludo/akhq:0.22.0
[0.003s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /app/hs_err_pid8.log
[root@akhq-abn app]#

Working:

[root@akhq-test akhq]# docker run tchiotludo/akhq:0.22.0
 __  __ _                                  _
|  \/  (_) ___ _ __ ___  _ __   __ _ _   _| |_
| |\/| | |/ __| '__/ _ \| '_ \ / _` | | | | __|
| |  | | | (__| | | (_) | | | | (_| | |_| | |_
|_|  |_|_|\___|_|  \___/|_| |_|\__,_|\__,_|\__|
  Micronaut (v3.2.2)

2022-09-09 12:03:23,072 ERROR main       i.m.runtime.Micronaut      Error starting Micronaut server: Failed to read configuration file: /app/application.yml
io.micronaut.context.exceptions.ConfigurationException: Failed to read configuration file: /app/application.yml
        at io.micronaut.context.env.DefaultEnvironment.readPropertiesFromLoader(DefaultEnvironment.java:613)
        at io.micronaut.context.env.DefaultEnvironment.lambda$readPropertySourceListFromFiles$5(DefaultEnvironment.java:467)
        at java.base/java.util.Optional.ifPresent(Unknown Source)
        at io.micronaut.context.env.DefaultEnvironment.readPropertySourceListFromFiles(DefaultEnvironment.java:455)
        at io.micronaut.context.env.DefaultEnvironment.readPropertySources(DefaultEnvironment.java:417)
        at io.micronaut.context.env.DefaultEnvironment.start(DefaultEnvironment.java:271)
        at io.micronaut.context.DefaultApplicationContext$RuntimeConfiguredEnvironment.start(DefaultApplicationContext.java:720)
        at io.micronaut.context.DefaultApplicationContext$RuntimeConfiguredEnvironment.start(DefaultApplicationContext.java:689)
        at io.micronaut.context.DefaultApplicationContext.startEnvironment(DefaultApplicationContext.java:230)
        at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:182)
        at io.micronaut.runtime.Micronaut.start(Micronaut.java:72)
        at io.micronaut.runtime.Micronaut.run(Micronaut.java:313)
        at io.micronaut.runtime.Micronaut.run(Micronaut.java:299)
        at org.akhq.App.main(App.java:27)
[root@akhq-test akhq]#

[jheinitz]

I cannot even run java -version:

[root@akhq-abn app]# docker run -it tchiotludo/akhq:0.22.0 bash
akhq@39baa461e506:/app$ java -version
[0.003s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /app/hs_err_pid10.log
akhq@39baa461e506:/app$ cat /app/hs_err_pid10.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
#   JVM is running with Unscaled Compressed Oops mode in which the Java heap is
#     placed in the first 4GB address space. The Java Heap base address is the
#     maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
#     to set the Java Heap base and to place the Java Heap above 4GB virtual address.
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (workerManager.hpp:70), pid=10, tid=10
#
# JRE version:  (11.0.16.1+1) (build )
# Java VM: OpenJDK 64-Bit Server VM (11.0.16.1+1, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /app/core.10)
#

---------------  S U M M A R Y ------------

Command Line:

Host: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz, 8 cores, 31G, Ubuntu 22.04.1 LTS
Time: Fri Sep  9 12:17:02 2022 UTC elapsed time: 0.003898 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread (0x00005592c0318800):  JavaThread "Unknown thread" [_thread_in_vm, id=10, stack(0x00007ffd2e1f6000,0x00007ffd2e9f4000)]

Stack: [0x00007ffd2e1f6000,0x00007ffd2e9f4000],  sp=0x00007ffd2e9eeb20,  free space=8162k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xf4720a]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1ca
V  [libjvm.so+0xf480cb]  VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x2b
V  [libjvm.so+0x6d3df5]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xd5
V  [libjvm.so+0xf7466b]  AbstractWorkGang::add_workers(unsigned int, bool)+0x14b
V  [libjvm.so+0x7afb55]  G1CollectedHeap::G1CollectedHeap(G1CollectorPolicy*)+0x5b5
V  [libjvm.so+0x7a1649]  G1Arguments::create_heap()+0x69
V  [libjvm.so+0xf02951]  Universe::initialize_heap()+0x31
V  [libjvm.so+0xf02c06]  universe_init()+0x46
V  [libjvm.so+0x8a9497]  init_globals()+0x57
V  [libjvm.so+0xeda277]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x357
V  [libjvm.so+0x9670d2]  JNI_CreateJavaVM+0x52
C  [libjli.so+0x49af]  JavaMain+0x8f
C  [libjli.so+0x9d94]  CallJavaMainInNewThread+0x74
C  [libjli.so+0x637d]  ContinueInNewThread+0x4d
C  [libjli.so+0x7a00]  JLI_Launch+0x1470
C  [java+0xa73]


---------------  P R O C E S S  ---------------

Threads class SMR info:
_java_thread_list=0x00005592c02e04f0, length=0, elements={
}

Java Threads: ( => current thread )

Other Threads:

=>0x00005592c0318800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=10, stack(0x00007ffd2e1f6000,0x00007ffd2e9f4000)]

Threads with active compile tasks:

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None


[error occurred during error reporting (printing compressed oops mode), id 0xb, SIGSEGV (0xb) at pc=0x00007f794a501283]

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (2 events):
Event: 0.002 Protecting memory [0x00007ffd2e1f6000,0x00007ffd2e1fa000] with protection modes 0
Event: 0.002 Loaded shared library /opt/java/openjdk/lib/libzip.so


Dynamic libraries:
5592bee7d000-5592bee7e000 r-xp 00000000 fd:04 34038238                   /opt/java/openjdk/bin/java
5592bf07e000-5592bf07f000 r--p 00001000 fd:04 34038238                   /opt/java/openjdk/bin/java
5592bf07f000-5592bf080000 rw-p 00002000 fd:04 34038238                   /opt/java/openjdk/bin/java
5592c02dc000-5592c034f000 rw-p 00000000 00:00 0                          [heap]
7f7933716000-7f7933ad3000 rw-p 00000000 00:00 0
7f7933ad3000-7f7933d43000 rwxp 00000000 00:00 0
7f7933d43000-7f7934064000 ---p 00000000 00:00 0
7f7934064000-7f79342d4000 rwxp 00000000 00:00 0
7f79342d4000-7f793b59b000 ---p 00000000 00:00 0
7f793b59b000-7f793b80b000 rwxp 00000000 00:00 0
7f793b80b000-7f7942ad3000 ---p 00000000 00:00 0
7f7942ad3000-7f7948600000 r--s 00000000 fd:04 421724                     /opt/java/openjdk/lib/modules
7f7948600000-7f7948607000 r-xp 00000000 fd:04 421723                     /opt/java/openjdk/lib/libzip.so
7f7948607000-7f7948807000 ---p 00007000 fd:04 421723                     /opt/java/openjdk/lib/libzip.so
7f7948807000-7f7948808000 r--p 00007000 fd:04 421723                     /opt/java/openjdk/lib/libzip.so
7f7948808000-7f7948809000 rw-p 00008000 fd:04 421723                     /opt/java/openjdk/lib/libzip.so
7f79488fe000-7f79488ff000 ---p 00000000 00:00 0
7f79488ff000-7f7948a00000 rw-p 00000000 00:00 0
7f7948a00000-7f7948a1b000 r-xp 00000000 fd:04 421706                     /opt/java/openjdk/lib/libjimage.so
7f7948a1b000-7f7948c1a000 ---p 0001b000 fd:04 421706                     /opt/java/openjdk/lib/libjimage.so
7f7948c1a000-7f7948c1c000 r--p 0001a000 fd:04 421706                     /opt/java/openjdk/lib/libjimage.so
7f7948c1c000-7f7948c1d000 rw-p 0001c000 fd:04 421706                     /opt/java/openjdk/lib/libjimage.so
7f7948d15000-7f7948d1a000 rw-p 00000000 00:00 0
7f7948d1a000-7f7948e00000 ---p 00000000 00:00 0
7f7948e00000-7f7948e28000 r-xp 00000000 fd:04 421702                     /opt/java/openjdk/lib/libjava.so
7f7948e28000-7f7949028000 ---p 00028000 fd:04 421702                     /opt/java/openjdk/lib/libjava.so
7f7949028000-7f7949029000 r--p 00028000 fd:04 421702                     /opt/java/openjdk/lib/libjava.so
7f7949029000-7f794902a000 rw-p 00029000 fd:04 421702                     /opt/java/openjdk/lib/libjava.so
7f794902a000-7f794902b000 rw-p 00000000 00:00 0
7f7949115000-7f794911a000 rw-p 00000000 00:00 0
7f794911a000-7f7949200000 ---p 00000000 00:00 0
7f7949200000-7f794920d000 r-xp 00000000 fd:04 421722                     /opt/java/openjdk/lib/libverify.so
7f794920d000-7f794940c000 ---p 0000d000 fd:04 421722                     /opt/java/openjdk/lib/libverify.so
7f794940c000-7f794940e000 r--p 0000c000 fd:04 421722                     /opt/java/openjdk/lib/libverify.so
7f794940e000-7f794940f000 rw-p 0000e000 fd:04 421722                     /opt/java/openjdk/lib/libverify.so
7f7949419000-7f7949519000 rw-p 00000000 00:00 0
7f7949519000-7f7949527000 r--p 00000000 fd:04 174727                     /usr/lib/x86_64-linux-gnu/libm.so.6
7f7949527000-7f79495a3000 r-xp 0000e000 fd:04 174727                     /usr/lib/x86_64-linux-gnu/libm.so.6
7f79495a3000-7f79495fe000 r--p 0008a000 fd:04 174727                     /usr/lib/x86_64-linux-gnu/libm.so.6
7f79495fe000-7f79495ff000 r--p 000e4000 fd:04 174727                     /usr/lib/x86_64-linux-gnu/libm.so.6
7f79495ff000-7f7949600000 rw-p 000e5000 fd:04 174727                     /usr/lib/x86_64-linux-gnu/libm.so.6
7f7949600000-7f794a876000 r-xp 00000000 fd:04 421731                     /opt/java/openjdk/lib/server/libjvm.so
7f794a876000-7f794aa76000 ---p 01276000 fd:04 421731                     /opt/java/openjdk/lib/server/libjvm.so
7f794aa76000-7f794ab3b000 r--p 01276000 fd:04 421731                     /opt/java/openjdk/lib/server/libjvm.so
7f794ab3b000-7f794ab76000 rw-p 0133b000 fd:04 421731                     /opt/java/openjdk/lib/server/libjvm.so
7f794ab76000-7f794abcd000 rw-p 00000000 00:00 0
7f794abd8000-7f794ac00000 r--p 00000000 fd:04 174802                     /usr/lib/x86_64-linux-gnu/libc.so.6
7f794ac00000-7f794ad95000 r-xp 00028000 fd:04 174802                     /usr/lib/x86_64-linux-gnu/libc.so.6
7f794ad95000-7f794aded000 r--p 001bd000 fd:04 174802                     /usr/lib/x86_64-linux-gnu/libc.so.6
7f794aded000-7f794adf1000 r--p 00214000 fd:04 174802                     /usr/lib/x86_64-linux-gnu/libc.so.6
7f794adf1000-7f794adf3000 rw-p 00218000 fd:04 174802                     /usr/lib/x86_64-linux-gnu/libc.so.6
7f794adf3000-7f794ae00000 rw-p 00000000 00:00 0
7f794ae00000-7f794ae10000 r-xp 00000000 fd:04 34038191                   /opt/java/openjdk/lib/jli/libjli.so
7f794ae10000-7f794b00f000 ---p 00010000 fd:04 34038191                   /opt/java/openjdk/lib/jli/libjli.so
7f794b00f000-7f794b010000 r--p 0000f000 fd:04 34038191                   /opt/java/openjdk/lib/jli/libjli.so
7f794b010000-7f794b011000 rw-p 00010000 fd:04 34038191                   /opt/java/openjdk/lib/jli/libjli.so
7f794b073000-7f794b078000 rw-p 00000000 00:00 0
7f794b078000-7f794b07f000 ---p 00000000 00:00 0
7f794b07f000-7f794b087000 rw-s 00000000 fd:04 34038121                   /tmp/hsperfdata_akhq/10
7f794b087000-7f794b088000 r--p 00000000 fd:04 174772                     /usr/lib/x86_64-linux-gnu/librt.so.1
7f794b088000-7f794b089000 r-xp 00001000 fd:04 174772                     /usr/lib/x86_64-linux-gnu/librt.so.1
7f794b089000-7f794b08a000 r--p 00002000 fd:04 174772                     /usr/lib/x86_64-linux-gnu/librt.so.1
7f794b08a000-7f794b08b000 r--p 00002000 fd:04 174772                     /usr/lib/x86_64-linux-gnu/librt.so.1
7f794b08b000-7f794b08c000 rw-p 00003000 fd:04 174772                     /usr/lib/x86_64-linux-gnu/librt.so.1
7f794b08c000-7f794b090000 rw-p 00000000 00:00 0
7f794b090000-7f794b091000 r--p 00000000 fd:04 174817                     /usr/lib/x86_64-linux-gnu/libdl.so.2
7f794b091000-7f794b092000 r-xp 00001000 fd:04 174817                     /usr/lib/x86_64-linux-gnu/libdl.so.2
7f794b092000-7f794b093000 r--p 00002000 fd:04 174817                     /usr/lib/x86_64-linux-gnu/libdl.so.2
7f794b093000-7f794b094000 r--p 00002000 fd:04 174817                     /usr/lib/x86_64-linux-gnu/libdl.so.2
7f794b094000-7f794b095000 rw-p 00003000 fd:04 174817                     /usr/lib/x86_64-linux-gnu/libdl.so.2
7f794b095000-7f794b096000 r--p 00000000 fd:04 174770                     /usr/lib/x86_64-linux-gnu/libpthread.so.0
7f794b096000-7f794b097000 r-xp 00001000 fd:04 174770                     /usr/lib/x86_64-linux-gnu/libpthread.so.0
7f794b097000-7f794b098000 r--p 00002000 fd:04 174770                     /usr/lib/x86_64-linux-gnu/libpthread.so.0
7f794b098000-7f794b099000 r--p 00002000 fd:04 174770                     /usr/lib/x86_64-linux-gnu/libpthread.so.0
7f794b099000-7f794b09a000 rw-p 00003000 fd:04 174770                     /usr/lib/x86_64-linux-gnu/libpthread.so.0
7f794b09a000-7f794b09c000 r--p 00000000 fd:04 173335                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f794b09c000-7f794b0ad000 r-xp 00002000 fd:04 173335                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f794b0ad000-7f794b0b3000 r--p 00013000 fd:04 173335                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f794b0b3000-7f794b0b4000 ---p 00019000 fd:04 173335                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f794b0b4000-7f794b0b5000 r--p 00019000 fd:04 173335                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f794b0b5000-7f794b0b6000 rw-p 0001a000 fd:04 173335                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f794b0b7000-7f794b0b8000 ---p 00000000 00:00 0
7f794b0b8000-7f794b0b9000 r--p 00000000 00:00 0
7f794b0b9000-7f794b0bb000 rw-p 00000000 00:00 0
7f794b0bb000-7f794b0bd000 r--p 00000000 fd:04 174784                     /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f794b0bd000-7f794b0e7000 r-xp 00002000 fd:04 174784                     /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f794b0e7000-7f794b0f2000 r--p 0002c000 fd:04 174784                     /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f794b0f3000-7f794b0f5000 r--p 00037000 fd:04 174784                     /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f794b0f5000-7f794b0f7000 rw-p 00039000 fd:04 174784                     /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ffd2e1f6000-7ffd2e1fa000 ---p 00000000 00:00 0
7ffd2e9d5000-7ffd2e9f6000 rw-p 00000000 00:00 0                          [stack]
7ffd2e9f8000-7ffd2e9fc000 r--p 00000000 00:00 0                          [vvar]
7ffd2e9fc000-7ffd2e9fe000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


VM Arguments:
java_command: <unknown>
java_class_path (initial): <not set>
Launcher Type: SUN_STANDARD

[Global flags]
     intx CICompilerCount                          = 4                                         {product} {ergonomic}
     uint G1ConcRefinementThreads                  = 8                                         {product} {ergonomic}
   size_t G1HeapRegionSize                         = 2097152                                   {product} {ergonomic}
    uintx GCDrainStackTargetSize                   = 64                                        {product} {ergonomic}
   size_t InitialHeapSize                          = 524288000                                 {product} {ergonomic}
   size_t MaxHeapSize                              = 8371830784                                {product} {ergonomic}
   size_t MinHeapDeltaBytes                        = 2097152                                   {product} {ergonomic}
    uintx NonNMethodCodeHeapSize                   = 5836300                                {pd product} {ergonomic}
    uintx NonProfiledCodeHeapSize                  = 122910970                              {pd product} {ergonomic}
    uintx ProfiledCodeHeapSize                     = 122910970                              {pd product} {ergonomic}
    uintx ReservedCodeCacheSize                    = 251658240                              {pd product} {ergonomic}
     bool SegmentedCodeCache                       = true                                      {product} {ergonomic}
     bool UseCompressedClassPointers               = true                                 {lp64_product} {ergonomic}
     bool UseCompressedOops                        = true                                 {lp64_product} {ergonomic}
     bool UseG1GC                                  = true                                      {product} {ergonomic}

Logging:
Log output configuration:
 #0: stdout all=warning uptime,level,tags
 #1: stderr all=off uptime,level,tags

Environment Variables:
JAVA_HOME=/opt/java/openjdk
PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8

Signal Handlers:
SIGSEGV: [libjvm.so+0xf48ea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xf48ea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0xf48ea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0xc6ad60], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0xc6ad60], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0xf48ea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR2: [libjvm.so+0xc6ac00], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none


---------------  S Y S T E M  ---------------

OS:DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
uname:Linux 4.18.0-348.el8.x86_64 #1 SMP Mon Oct 4 12:17:22 EDT 2021 x86_64
OS uptime: 119 days 22:59 hours
libc:glibc 2.35 NPTL 2.35
rlimit (soft/hard): STACK 8192k/infinity , CORE infinity/infinity , NPROC infinity/infinity , NOFILE 1048576/1048576 , AS infinity/infinity , CPU infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity , MEMLOCK 64k/64k
load average:0.26 0.24 0.27

/proc/meminfo:
MemTotal:       32700176 kB
MemFree:         7865528 kB
MemAvailable:   20200360 kB
Buffers:            3432 kB
Cached:         12503340 kB
SwapCached:         3040 kB
Active:          1544156 kB
Inactive:       22363420 kB
Active(anon):     199516 kB
Inactive(anon): 11202196 kB
Active(file):    1344640 kB
Inactive(file): 11161224 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2097148 kB
SwapFree:        1998332 kB
Dirty:              1752 kB
Writeback:             0 kB
AnonPages:      10499828 kB
Mapped:           460452 kB
Shmem:               856 kB
KReclaimable:     294072 kB
Slab:             522068 kB
SReclaimable:     294072 kB
SUnreclaim:       227996 kB
KernelStack:       15920 kB
PageTables:        40976 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    18447236 kB
Committed_AS:   20668892 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:           155648 kB
HardwareCorrupted:     0 kB
AnonHugePages:   7161856 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      573312 kB
DirectMap2M:    32980992 kB

/sys/kernel/mm/transparent_hugepage/enabled:
[always] madvise never
/sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter):
always defer defer+madvise [madvise] never

Process Memory:
Virtual Size: 384200K (peak: 384200K)
Resident Set Size: 18260K (peak: 18260K) (anon: 5484K, file: 12776K, shmem: 0K)
Swapped out: 0K
C-Heap outstanding allocations: 4217K, retained: 70K
glibc malloc tunables: (default)

/proc/sys/kernel/threads-max (system-wide limit on the number of threads):
255156
/proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):
65530
/proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):
4194304

container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-7
cpu_memory_nodes: 0
active_processor_count: 8
cpu_quota: no quota
cpu_period: 100000
cpu_shares: no shares
memory_limit_in_bytes: unlimited
memory_and_swap_limit_in_bytes: unlimited
memory_soft_limit_in_bytes: unlimited
memory_usage_in_bytes: 7888896
memory_max_usage_in_bytes: 9564160

VMWare virtualization detected
Steal ticks since vm start: 0
Steal ticks percentage since vm start:  0.000

CPU:total 8 (initial active 8) (1 cores per cpu, 1 threads per core) family 6 model 62 stepping 4 microcode 0x42e, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, tsc, tscinvbit
CPU Model and flags from /proc/cpuinfo:
model name      : Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust smep arat md_clear flush_l1d arch_capabilities
Online cpus:
0-7
Offline cpus:
8-127
BIOS frequency limitation:
<Not Available>
Frequency switch latency (ns):
<Not Available>
Available cpu frequencies:
<Not Available>
Current governor:
<Not Available>
Core performance/turbo boost:
<Not Available>

Memory: 4k page, physical 32700176k(7865528k free), swap 2097148k(1998332k free)

vm_info: OpenJDK 64-Bit Server VM (11.0.16.1+1) for linux-amd64 JRE (11.0.16.1+1), built on Aug 16 2022 07:24:27 by "" with gcc 7.5.0

END.
akhq@39baa461e506:/app$

[tchiotludo]

what is the os that is running docker?

[jheinitz]

It is RedHat on both. The one that is working is on RedHat 8.6, the one that is NOT working is on RedHat 8.5. Both have a different docker version:

• not working: 20.10.9
• working: 20.10.17

[jheinitz]

I just found this. Are you using jammy or focal?

[tchiotludo]

If I understand here, it's jammy

Just confirmed inside the container

root@5abda67f6424:/etc# lsb_release -a 
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy

[jheinitz]

Hello @erkexzcx !

Were you able to make any progress on this issue?

Thanks and kind regards

Jens

[jheinitz]

If I understand here, it's jammy

Just confirmed inside the container

root@5abda67f6424:/etc# lsb_release -a 
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy

Would it be possible to provide an AKQ image based on the focal instead of jammy?

KR
Jens

[erkexzcx]

Hello @erkexzcx !

Were you able to make any progress on this issue?

Thanks and kind regards

Jens

Yes.,? I downgraded docker image from 0.22.0 to 0.21.0 and everything runs great again.

[jheinitz]

Hello @erkexzcx !
Were you able to make any progress on this issue?
Thanks and kind regards
Jens

Yes.,? I downgraded docker image from 0.22.0 to 0.21.0 and everything runs great again.

This works for me as well, but I'd like to have the features of 0.22.0. That's why I would like to get this issue solved.
Thanks for answering so fast.

Best regards
Jens

[tchiotludo]

Would it be possible to provide an AKQ image based on the focal instead of jammy?

You could easily built it locally and see if the issue is resolved or not ?

It's clearly not the goal to use old version instead of new one, I would prefer understand the issue and fix it but I'm not able to reproduce it at all.

[evaisman-tc]

It is RedHat on both. The one that is working is on RedHat 8.6, the one that is NOT working is on RedHat 8.5. Both have a different docker version:

* not working: 20.10.9

* working: 20.10.17

I'm experiencing similar issues with the 0.22.0 image on Debian 10 with docker version 18.09.5

    Distributor ID:	Debian
    Description:	Debian GNU/Linux 10 (buster)
    Release:	10
    Codename:	buster

[Maxsky5]

Same issue on Ubuntu 20 with version 0.22.0 :

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

[jheinitz]

Hi,

it seems to be a problem with the docker version. I can confirm for RedHat that it works on RedHat 8.6 with a newer docker version than on RedHat 8.5. Please see this comment for details.

Cheers

Jens

[tchiotludo]

The issue is on docker side and I can't do anything.
Please upgrade docker version