openssl 1.1.1k incompatibilities

[leogr]

Describe the bug

After upgrading openssl from 1.0.2n to 1.1.1k, a program freeze may occur when using the K8s client implementation provided by sinsp. I have encountered this problem while testing the latest (at the time of writing) Falco dev version 0.29.1-28+0eb170c which comes with a driver version equal to: 5727c45

To reproduce the bug, I have used the helm chart with the following docker image (which includes Falco 0.29.1-28+0eb170c):

falcosecurity/falco@sha256:e0f81a84c9b90c084f9dfef4443e3f8877784cca9ce2008d8324fc9e75b7c89b

At the time of writing, it was tagged as falcosecurity/falco:master.

The problem occurs only when the k8s support is enabled and Falco is running in a container. Strangely, the bug does not occur when using Falco on the host directly.

How to reproduce it

helm install falco falcosecurity/falco --set image.tag=master --set fakeEventGenerator.enabled=enabled

Then Falco immediately freezes and no alerts are emitted.

Expected behaviour

No freeze.

Screenshots

Environment

• Falco version: Falco version 0.29.1-28+0eb170c (driver version 5727c456ce22f3c1da8c3b1d7d6b6937a9b2126b)
• System info:

{
  "machine": "x86_64",
  "nodename": "falco-ktllt",
  "release": "5.14.8-arch1-1",
  "sysname": "Linux",
  "version": "#1 SMP PREEMPT Sun, 26 Sep 2021 19:36:15 +0000"
}

• Cloud provider or hardware configuration: kind on my local machine
• OS: ArchLinux
• Kernel: Linux x86 5.14.8-arch1-1 #1 SMP PREEMPT Sun, 26 Sep 2021 19:36:15 +0000 x86_64 GNU/Linux
• Installation method: helm

Additional context

The bug was introduced by 16c7aef

A working patch (ie. downgrading openssl to the 1.0.x series) is available here #90

This problem is blocking the Falco 0.30.0 release. We are working on a hotfix.

[leogr]

🤔

(gdb) bt
#0  sccp (nr=<optimized out>, u=140737488342704, v=140737488342704, w=0, x=0, y=0, z=0) at ./arch/x86_64/syscall_arch.h:61
#1  0x0000000000d8ceec in __clock_nanosleep (clk=clk@entry=0, flags=flags@entry=0, req=req@entry=0x7fffffffceb0, rem=rem@entry=0x7fffffffceb0) at src/time/clock_nanosleep.c:34
#2  0x0000000000d853ad in nanosleep (req=req@entry=0x7fffffffceb0, rem=rem@entry=0x7fffffffceb0) at src/time/nanosleep.c:6
#3  0x0000000000d8679e in usleep (useconds=<optimized out>) at src/unistd/usleep.c:11
#4  0x00000000004fbcd6 in socket_data_handler<k8s_handler>::get_all_data (this=0x7fffdf0c7ac0) at /usr/include/c++/10.3.1/ext/new_allocator.h:89
#5  0x00000000004f1a7b in k8s_handler::receive_response (this=this@entry=0x7fffdef37f50)
    at /source/build-musl/falcosecurity-libs-repo/falcosecurity-libs-prefix/src/falcosecurity-libs/userspace/libsinsp/k8s_handler.cpp:276
#6  0x00000000004f5716 in k8s_handler::collect_data (this=0x7fffdef37f50)
    at /source/build-musl/falcosecurity-libs-repo/falcosecurity-libs-prefix/src/falcosecurity-libs/userspace/libsinsp/k8s_handler.cpp:413
#7  0x00000000004d4473 in sinsp::update_k8s_state (this=this@entry=0x7ffff7fe27e0)
    at /source/build-musl/falcosecurity-libs-repo/falcosecurity-libs-prefix/src/falcosecurity-libs/userspace/libsinsp/sinsp.cpp:2233
#8  0x00000000004d49f5 in sinsp::next (this=0x7ffff7fe27e0, puevt=0x7fffffffd738)
    at /source/build-musl/falcosecurity-libs-repo/falcosecurity-libs-prefix/src/falcosecurity-libs/userspace/libsinsp/sinsp.cpp:1257
#9  0x0000000000448ddd in do_inspect (engine=engine@entry=0x7ffff7fcb550, outputs=outputs@entry=0x7ffff7438e30, inspector=inspector@entry=0x7ffff7fe27e0, config=..., sdropmgr=..., 
    duration_to_tot_ns=duration_to_tot_ns@entry=0, stats_filename=..., stats_interval=5000, all_events=false, result=@0x7fffffffdb98: 0) at /source/userspace/falco/falco.cpp:285
#10 0x000000000044c6c7 in falco_init (argc=argc@entry=8, argv=argv@entry=0x7fffffffeb48) at /source/userspace/falco/falco.cpp:1357
#11 0x000000000042d1da in main (argc=8, argv=0x7fffffffeb48) at /source/userspace/falco/falco.cpp:1440

[IanRobertson-wpe]

I'm not sure if this impacts Falco or its users, but I figured I'd call out that 1.0.2u may exhibit issues with the expiring Let's Encrypt root certificates (which happens tomorrow): https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/.

Also 1.0.2u has several CVEs, which will make the new Falco image immediately show up as vulnerable. https://www.openssl.org/news/vulnerabilities.html

[leogr]

Hi @IanRobertson-wpe

Thanks for reporting that! It is certainly something we need to take into consideration.

Although not all OpenSSL CVEs may directly affect Falco (since not all OpenSSL features are used in sinsp) and its users, I generally agree with you. This is an unfortunate situation.

Anyway, after a further investigation, we are opting to downgrade OpenSSL to 1.0.2u (which is still newer than the last one shipped with Falco, i.e., OpenSSL 1.0.2n) because we discovered several incompatibilities with the 1.1.x series which cannot be solved shortly. These incompatibilities can cause significant and more severe issues to Falco users.

For sure, we have to address the OpenSSL upgrading issue soon after the upcoming release (Falco 0.30.0, it should come out tomorrow). We may also decide to release a patch version, eventually, without having to wait until Falco 0.31.0 in Jan.