Revert ext_proc: send attributes #30781 #31017

yanjunxiang-google · 2023-11-22T15:14:24Z

Description:

After the commit of #30781 , ext_proc filter unit test fuzzer is reporting crashes.
We suspect there is lifetime issues in the cel expression initialization code during filter config parsing. The crash can be simply reproduced by below command with the two crash input files committed with this PR:

bazel test --config asan-fuzzer //test/extensions/filters/http/ext_proc/unit_test_fuzz:ext_proc_unit_test_fuzz --test_timeout=300

Revert the PR for now.

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
Fixes commit #30781
[Optional Deprecated:]
[Optional API Considerations:]

Signed-off-by: Yanjun Xiang <[email protected]>

repokitteh-read-only · 2023-11-22T15:14:35Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @wbpcode
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #31017 was opened by yanjunxiang-google.

see: more, trace.

yanjunxiang-google · 2023-11-22T15:17:51Z

/assign @jbohanon @tyxia @yanavlasov

repokitteh-read-only · 2023-11-22T15:17:56Z

@jbohanon cannot be assigned to this issue.

🐱

Caused by: a #31017 (comment) was created by @yanjunxiang-google.

see: more, trace.

yanjunxiang-google · 2023-11-22T15:19:22Z

This is the ASAN crash traceback decode:

=================================================================
==3697==ERROR: AddressSanitizer: container-overflow on address 0x60c00002e1c8 at pc 0x000001c3031b bp 0x7fffd5258ed0 sp 0x7fffd5258ec8
READ of size 8 at 0x60c00002e1c8 thread T0
#0 0x1c3031a in reset /proc/self/cwd/external/antlr4_runtimes/runtime/Cpp/runtime/src/tree/ParseTree.h:100:25
#1 0x1c3031a in antlr4::Parser::~Parser() /proc/self/cwd/external/antlr4_runtimes/runtime/Cpp/runtime/src/Parser.cpp:99:12
#2 0x1c04cfd in cel_parser_internal::CelParser::~CelParser() /proc/self/cwd/bazel-out/k8-fastbuild-ST-9a5eb34baa51/bin/external/com_google_cel_cpp/parser/internal/CelParser.cpp:162:1
#3 0x1b77a55 in google::api::expr::parser::EnrichedParse(std::__1::basic_string_view<char, std::__1::char_traits >, std::__1::vector<cel::Macro, std::__1::allocatorcel::Macro > const&, std::__1::basic_string_view<char, std::__1::char_traits >, cel::ParserOptions const&) /proc/self/cwd/external/com_google_cel_cpp/parser/parser.cc:1292:3
#4 0x1b7179d in google::api::expr::parser::ParseWithMacros(std::__1::basic_string_view<char, std::__1::char_traits >, std::__1::vector<cel::Macro, std::__1::allocatorcel::Macro > const&, std::__1::basic_string_view<char, std::__1::char_traits >, cel::ParserOptions const&) /proc/self/cwd/external/com_google_cel_cpp/parser/parser.cc:1236:3
#5 0x1b71540 in google::api::expr::parser::Parse(std::__1::basic_string_view<char, std::__1::char_traits >, std::__1::basic_string_view<char, std::__1::char_traits >, cel::ParserOptions const&) /proc/self/cwd/external/com_google_cel_cpp/parser/parser.cc:1229:10
#6 0x1aba2e3 in Envoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig::initExpressions(google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > const&) const /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.cc:125:25
#7 0x199a318 in Envoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig::FilterConfig(envoy::extensions::filters::http::ext_proc::v3::ExternalProcessor const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, unsigned int, Envoy::Stats::Scope&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptrEnvoy::Extensions::Filters::Common::Expr::BuilderInstance) /proc/self/cwd/source/extensions/filters/http/ext_proc/ext_proc.h:142:42
#8 0x1999bff in std::__1::__shared_ptr_emplace<Envoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig, std::__1::allocatorEnvoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig >::__shared_ptr_emplace<envoy::extensions::filters::http::ext_proc::v3::ExternalProcessor const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, int, Envoy::Stats::Scope&, char const (&) [16], std::__1::shared_ptrEnvoy::Extensions::Filters::Common::Expr::BuilderInstance >(std::__1::allocatorEnvoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig, envoy::extensions::filters::http::ext_proc::v3::ExternalProcessor const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&&, int&&, Envoy::Stats::Scope&, char const (&) [16], std::__1::shared_ptrEnvoy::Extensions::Filters::Common::Expr::BuilderInstance&&) /usr/local/include/c++/v1/__memory/shared_ptr.h:294:37
#9 0x18cf561 in allocate_shared<Envoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig, std::__1::allocatorEnvoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig, const envoy::extensions::filters::http::ext_proc::v3::ExternalProcessor &, std::__1::chrono::duration<long long, std::__1::ratio<1L, 1000L> >, int, Envoy::Stats::Scope &, const char (&)[16], std::__1::shared_ptrEnvoy::Extensions::Filters::Common::Expr::BuilderInstance, void> /usr/local/include/c++/v1/__memory/shared_ptr.h:953:55
#10 0x18cf561 in make_shared<Envoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig, const envoy::extensions::filters::http::ext_proc::v3::ExternalProcessor &, std::__1::chrono::duration<long long, std::__1::ratio<1L, 1000L> >, int, Envoy::Stats::Scope &, const char (&)[16], std::__1::shared_ptrEnvoy::Extensions::Filters::Common::Expr::BuilderInstance, void> /usr/local/include/c++/v1/__memory/shared_ptr.h:962:12
#11 0x18cf561 in TestOneProtoInput /proc/self/cwd/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz.cc:73:14
#12 0x18cf561 in LLVMFuzzerTestOneInput /proc/self/cwd/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz.cc:51:1
#13 0x17a27d3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15
#14 0x178d2e2 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
#15 0x1792b8c in fuzzer::FuzzerDriver(int*, char***, int ()(unsigned char const, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:860:9
#16 0x17bcd12 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
#17 0x7c97477af082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16
#18 0x17834ad in _start

jbohanon · 2023-11-22T15:21:53Z

Is this not a part of a required check? Is there a way to diagnose the test failure without reverting the feature immediately? What is the blast radius of the issue that the fuzz test is revealing?

tyxia

LGTM, Thanks

@jbohanon I left comment here https://github.com/envoyproxy/envoy/pull/30781/files#r1402254604.

It is probably not related to the specific crash here but I think it is wrong pattern of using CEL . Also, given it is close to holiday, it is a good idea that we revert it first, investigate the issue in the meantime.

yanjunxiang-google · 2023-11-22T16:00:15Z

Is this not a part of a required check? Is there a way to diagnose the test failure without reverting the feature immediately? What is the blast radius of the issue that the fuzz test is revealing?

We are a little bit concerned this may cause Envoy crash if someone accidentally configured request_attributes. So reverting it is safer.

Once we have a good root cause and fix, we can commit them back in.

wbpcode · 2023-11-23T02:49:49Z

#30781 was merged at 2 days ago. So, it is ok to revert it.

/lgtm api

wbpcode · 2023-11-23T02:50:46Z

/assign @yanavlasov for the code change.

repokitteh-read-only · 2023-11-23T02:50:53Z

🙀 Error while processing event:

evaluation error
error: function _rk_error error: path contains forbidden characters:
 Traceback (most recent call last):
  bootstrap:143: in <toplevel>
  bootstrap:135: in _main
  bootstrap:62: in _call
  bootstrap:15: in _call1
  github.com/repokitteh/modules/assign.star:18: in _assign
  github:395: in issue_check_assignee
  github:131: in call
Error: function _rk_error error: path contains forbidden characters

🐱

Caused by: a #31017 (comment) was created by @wbpcode.

see: more, trace.

yanavlasov · 2023-11-27T16:40:07Z

The reason to revert is that invalid config can cause data plane crash. This is to risky to keep this PR. It seems like validation of CEL expression is missing, but I'm not sure. It is better to investigate the crash without having added risk in the codebase.

yanavlasov · 2023-11-27T16:40:32Z

Waiting for the merge conflict to be resolved.

/wait

tyxia · 2023-11-27T16:50:17Z

The reason to revert is that invalid config can cause data plane crash. This is to risky to keep this PR. It seems like validation of CEL expression is missing, but I'm not sure. It is better to investigate the crash without having added risk in the codebase.

AFAIU, it is not invalid config but the fuzzer issue instead? cced @adisuissa who figured out this.

Back to original CL, my 2 cents is that we should address the issue i mentioned before merging it.

Signed-off-by: Yanjun Xiang <[email protected]>

adisuissa · 2023-11-27T18:18:29Z

The reason to revert is that invalid config can cause data plane crash. This is to risky to keep this PR. It seems like validation of CEL expression is missing, but I'm not sure. It is better to investigate the crash without having added risk in the codebase.

AFAIU, it is not invalid config but the fuzzer issue instead? cced @adisuissa who figured out this.

The fuzz-issue was fixed in google/oss-fuzz#11258 by enabling instrumentation in the CEL code.
AFAIK there are no open fuzz issues related to ext_proc and CEL.

yanjunxiang-google · 2023-11-27T18:40:07Z

The reason to revert is that invalid config can cause data plane crash. This is to risky to keep this PR. It seems like validation of CEL expression is missing, but I'm not sure. It is better to investigate the crash without having added risk in the codebase.

AFAIU, it is not invalid config but the fuzzer issue instead? cced @adisuissa who figured out this.

The fuzz-issue was fixed in google/oss-fuzz#11258 by enabling instrumentation in the CEL code. AFAIK there are no open fuzz issues related to ext_proc and CEL.

So, with the fix of google/oss-fuzz#11258, the ext_proc fuzz crash is still observed with the change of #30781, right?

adisuissa · 2023-11-27T18:43:13Z

So, with the fix of google/oss-fuzz#11258, the ext_proc fuzz crash is still observed with the change of #30781, right?

Which ext_proc fuzz crash are you referring to? Can you paste the link?

yanjunxiang-google · 2023-11-27T19:01:20Z

So, with the fix of google/oss-fuzz#11258, the ext_proc fuzz crash is still observed with the change of #30781, right?

Which ext_proc fuzz crash are you referring to? Can you paste the link?

I updated my workspace to latest, and in the process of running the ext_proc unit test fuzzer and see whether there is any crash in ext_proc code.

adisuissa · 2023-11-27T19:04:36Z

I am running the ext_proc unit test fuzzer and see whether there is any crash in ext_proc code.

This is not an oss-fuzz crash, and I believe it has to do something with the build flags that are used for asan-fuzzer compared with asan. Unfortunately I don't have cycles to debug what are the flags that cause this, but "oss-fuzz" should be considered as the source-of-truth. Do you have cycles to debug what is causing this?

yanjunxiang-google · 2023-11-27T19:45:01Z

With the latest workspace which includes google/oss-fuzz#11258, but does not include this revert PR , I saw the crash below Envoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig::initExpressions(). Note, this function is added by #30781 , which is the reason why we are trying to revert it.

==12==ERROR: AddressSanitizer: heap-use-after-free on address 0x61100029ce00 at pc 0x00000341b53f bp 0x7fff612a6310 sp 0x7fff612a6308
WRITE of size 8 at 0x61100029ce00 thread T0
#0 0x341b53e in antlr4::RecognitionException::~RecognitionException() (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/ex
tensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x341b53e)
#1 0x33ed8a0 in antlr4::InputMismatchException::~InputMismatchException() (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/tes
t/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x33ed8a0)
#2 0x7fc56f2cbbee in __cxa_decrement_exception_refcount (/lib/x86_64-linux-gnu/libc++abi.so.1+0x27bee) (BuildId: 16898b48642f9766bffea0254fdfcd0d51f33700)
#3 0x33a3970 in cel_parser_internal::CelParser::unary() (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions/filte
rs/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x33a3970)
#4 0x339ec31 in cel_parser_internal::CelParser::calc(int) (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions/fil
ters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x339ec31)
#5 0x339c5a0 in cel_parser_internal::CelParser::relation(int) (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions
/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x339c5a0)
#6 0x3399e48 in cel_parser_internal::CelParser::conditionalAnd() (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensi
ons/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x3399e48)
#7 0x33978d6 in cel_parser_internal::CelParser::conditionalOr() (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensio
ns/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x33978d6)
#8 0x3395832 in cel_parser_internal::CelParser::expr() (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions/filter
s/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x3395832)
#9 0x3394c4e in cel_parser_internal::CelParser::start() (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions/filte
rs/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x3394c4e)
#10 0x32bc47f in google::api::expr::parser::EnrichedParse(std::__1::basic_string_view<char, std::__1::char_traits >, std::__1::vector<cel::Macro, std::__1::allocatorcel::Macro > const&, std::__1::basic_string_vie
w<char, std::__1::char_traits >, cel::ParserOptions const&) (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions
/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x32bc47f)
#11 0x32ba37d in google::api::expr::parser::ParseWithMacros(std::__1::basic_string_view<char, std::__1::char_traits >, std::__1::vector<cel::Macro, std::__1::allocatorcel::Macro > const&, std::__1::basic_string_v
iew<char, std::__1::char_traits >, cel::ParserOptions const&) (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensio
ns/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x32ba37d)
#12 0x32ba121 in google::api::expr::parser::Parse(std::__1::basic_string_view<char, std::__1::char_traits >, std::__1::basic_string_view<char, std::__1::char_traits >, cel::ParserOptions const&) (/usr/local/g
oogle/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x32ba121)
#13 0x319d07a in Envoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig::initExpressions(google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > \

const&) const (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_uni
t_test_fuzz+0x319d07a)
#14 0x308d940 in Envoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig::FilterConfig(envoy::extensions::filters::http::ext_proc::v3::ExternalProcessor const&, std::__1::chrono::duration<long long, std::__1::ra
tio<1l, 1000l> >, unsigned int, Envoy::Stats::Scope&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptr<Envoy::Extensions::Filters::Common::Expr::BuilderInstan
ce>) (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz
+0x308d940)
#15 0x308d377 in std::__1::__shared_ptr_emplace<Envoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig, std::__1::allocatorEnvoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig >::__shared_ptr_empl
ace<envoy::extensions::filters::http::ext_proc::v3::ExternalProcessor const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, int, Envoy::Stats::Scope&, char const (&) [16], std::__1::shared_ptr<Envoy::Ex
tensions::Filters::Common::Expr::BuilderInstance> >(std::__1::allocatorEnvoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig, envoy::extensions::filters::http::ext_proc::v3::ExternalProcessor const&, std::__1::c
hrono::duration<long long, std::__1::ratio<1l, 1000l> >&&, int&&, Envoy::Stats::Scope&, char const (&) [16], std::__1::shared_ptrEnvoy::Extensions::Filters::Common::Expr::BuilderInstance&&) (/usr/local/google/home/yanjunxi
ang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz+0x308d377)
#16 0x308cd90 in std::__1::shared_ptrEnvoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig std::__1::allocate_shared<Envoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig, std::__1::allocator<Envo
y::Extensions::HttpFilters::ExternalProcessing::FilterConfig>, envoy::extensions::filters::http::ext_proc::v3::ExternalProcessor const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, int, Envoy::Stats::
Scope&, char const (&) [16], std::__1::shared_ptrEnvoy::Extensions::Filters::Common::Expr::BuilderInstance, void>(std::__1::allocatorEnvoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig const&, envoy::extensi
ons::filters::http::ext_proc::v3::ExternalProcessor const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&&, int&&, Envoy::Stats::Scope&, char const (&) [16], std::__1::shared_ptr<Envoy::Extensions::Filt
ers::Common::Expr::BuilderInstance>&&) ext_proc_unit_test_fuzz.cc
#17 0x2f9be90 in std::__1::shared_ptrEnvoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig std::__1::make_shared<Envoy::Extensions::HttpFilters::ExternalProcessing::FilterConfig, envoy::extensions::filters::
http::ext_proc::v3::ExternalProcessor const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >, int, Envoy::Stats::Scope&, char const (&) [16], std::__1::shared_ptr<Envoy::Extensions::Filters::Common::Expr:
:BuilderInstance>, void>(envoy::extensions::filters::http::ext_proc::v3::ExternalProcessor const&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >&&, int&&, Envoy::Stats::Scope&, char const (&) [16], std::
__1::shared_ptrEnvoy::Extensions::Filters::Common::Expr::BuilderInstance&&) ext_proc_unit_test_fuzz.cc
#18 0x2f95314 in LLVMFuzzerTestOneInput (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions/filters/http/ext_proc
/unit_test_fuzz/ext_proc_unit_test_fuzz+0x2f95314)
#19 0x2e5c443 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) cxa_noexception.cpp
#20 0x2e5bc2a in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) cxa_noexception.cpp
#21 0x2e5d2f9 in fuzzer::Fuzzer::MutateAndTestOne() cxa_noexception.cpp
#22 0x2e5dfc5 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocatorfuzzer::SizedFile >&) cxa_noexception.cpp
#23 0x2e4c65f in fuzzer::FuzzerDriver(int*, char***, int ()(unsigned char const, unsigned long)) cxa_noexception.cpp
#24 0x2e769a2 in main (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/ex
t_proc_unit_test_fuzz+0x2e769a2)
#25 0x7fc56ec656c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#26 0x7fc56ec65784 in __libc_start_main csu/../csu/libc-start.c:360:3
#27 0x2e3d060 in _start (/usr/local/google/home/yanjunxiang/.cache/bazel/_bazel_yanjunxiang/81b91fea7a3a103a98a3bfda669fc1d9/execroot/envoy/bazel-out/k8-fastbuild/bin/test/extensions/filters/http/ext_proc/unit_test_fuzz/
ext_proc_unit_test_fuzz+0x2e3d060)

yanjunxiang-google · 2023-11-27T19:51:30Z

I am running the ext_proc unit test fuzzer and see whether there is any crash in ext_proc code.

This is not an oss-fuzz crash, and I believe it has to do something with the build flags that are used for asan-fuzzer compared with asan. Unfortunately I don't have cycles to debug what are the flags that cause this, but "oss-fuzz" should be considered as the source-of-truth. Do you have cycles to debug what is causing this?

I suspect there is some variable life-time issues in this function: FilterConfig::initExpressions, as @tyxia pointed out earlier. But I could be wrong. We should debug this after reverting this PR first.

jbohanon · 2023-11-30T20:04:56Z

This appears to be an issue with the upstream CEL parser. The fuzzer ASAN heap-use-after-free crash can be replicated without any net-new code by just calling Parse in the fuzz test body over the request attributes. This is demonstrated in this branch, where the output of bazel test --config asan-fuzzer //test/extensions/filters/http/ext_proc/unit_test_fuzz:ext_proc_unit_test_fuzz --test_timeout=300 is the same as above.

adisuissa · 2023-11-30T20:39:04Z

I think it is an issue with Envoy's building flags when using asan-fuzzer and calling CEL's Parser functions.
FWIW, it is not reproducible in oss-fuzz.

jbohanon · 2023-11-30T21:22:18Z

@yanavlasov @yanjunxiang-google @tyxia since this failure is unrelated to net-new code would you mind looking at #31090

yanjunxiang-google · 2023-11-30T21:54:56Z

ext_proc unit test fuzzer is the main one protect ext_proc code. If we commit this back in now, it will leave the fuzzer in broken state.

Can we spend a little bit more time on the root cause, or at least skip the asan-fuzzer test if request_attributes is configured in the fuzzer input?

envoy/test/extensions/filters/http/ext_proc/unit_test_fuzz/ext_proc_unit_test_fuzz.cc

Line 52 in d4ed8d4

    
           const envoy::extensions::filters::http::ext_proc::unit_test_fuzz::ExtProcUnitTestCase& input) {

jbohanon · 2023-12-01T14:27:56Z

@yanjunxiang-google in that PR I have added the conditional to skip the fuzzing of request_attributes and response_attributes if running --config asan-fuzzer.

I have confirmed the behavior change with the command listed above now timing out instead of failing with the previous error.

Revert ext_proc: send attributes envoyproxy#30781

7101e70

Signed-off-by: Yanjun Xiang <[email protected]>

yanjunxiang-google requested review from mattklein123, htuch and yanavlasov as code owners November 22, 2023 15:14

repokitteh-read-only bot added the api label Nov 22, 2023

repokitteh-read-only bot assigned wbpcode Nov 22, 2023

tyxia approved these changes Nov 22, 2023

View reviewed changes

botengyao approved these changes Nov 22, 2023

View reviewed changes

repokitteh-read-only bot removed the api label Nov 23, 2023

wbpcode assigned yanavlasov Nov 23, 2023

repokitteh-read-only bot added the waiting label Nov 27, 2023

merge upstream main

9196753

Signed-off-by: Yanjun Xiang <[email protected]>

repokitteh-read-only bot removed the waiting label Nov 27, 2023

repokitteh-read-only bot added the waiting label Nov 27, 2023

yanavlasov approved these changes Nov 28, 2023

View reviewed changes

yanavlasov merged commit 7730f8b into envoyproxy:main Nov 28, 2023
106 checks passed

jbohanon mentioned this pull request Nov 28, 2023

ext_proc: send attributes #31090

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert ext_proc: send attributes #30781 #31017

Revert ext_proc: send attributes #30781 #31017

yanjunxiang-google commented Nov 22, 2023 •

edited by yanavlasov

Loading

repokitteh-read-only bot commented Nov 22, 2023

yanjunxiang-google commented Nov 22, 2023

repokitteh-read-only bot commented Nov 22, 2023

yanjunxiang-google commented Nov 22, 2023 •

edited

Loading

jbohanon commented Nov 22, 2023

tyxia left a comment •

edited

Loading

yanjunxiang-google commented Nov 22, 2023

wbpcode commented Nov 23, 2023

wbpcode commented Nov 23, 2023

repokitteh-read-only bot commented Nov 23, 2023

yanavlasov commented Nov 27, 2023

yanavlasov commented Nov 27, 2023

tyxia commented Nov 27, 2023 •

edited

Loading

adisuissa commented Nov 27, 2023

yanjunxiang-google commented Nov 27, 2023

adisuissa commented Nov 27, 2023

yanjunxiang-google commented Nov 27, 2023 •

edited

Loading

adisuissa commented Nov 27, 2023

yanjunxiang-google commented Nov 27, 2023 •

edited

Loading

yanjunxiang-google commented Nov 27, 2023

jbohanon commented Nov 30, 2023 •

edited

Loading

adisuissa commented Nov 30, 2023

jbohanon commented Nov 30, 2023

yanjunxiang-google commented Nov 30, 2023 •

edited

Loading

jbohanon commented Dec 1, 2023

Revert ext_proc: send attributes #30781 #31017

Revert ext_proc: send attributes #30781 #31017

Conversation

yanjunxiang-google commented Nov 22, 2023 • edited by yanavlasov Loading

repokitteh-read-only bot commented Nov 22, 2023

yanjunxiang-google commented Nov 22, 2023

repokitteh-read-only bot commented Nov 22, 2023

yanjunxiang-google commented Nov 22, 2023 • edited Loading

jbohanon commented Nov 22, 2023

tyxia left a comment • edited Loading

Choose a reason for hiding this comment

yanjunxiang-google commented Nov 22, 2023

wbpcode commented Nov 23, 2023

wbpcode commented Nov 23, 2023

repokitteh-read-only bot commented Nov 23, 2023

yanavlasov commented Nov 27, 2023

yanavlasov commented Nov 27, 2023

tyxia commented Nov 27, 2023 • edited Loading

adisuissa commented Nov 27, 2023

yanjunxiang-google commented Nov 27, 2023

adisuissa commented Nov 27, 2023

yanjunxiang-google commented Nov 27, 2023 • edited Loading

adisuissa commented Nov 27, 2023

yanjunxiang-google commented Nov 27, 2023 • edited Loading

yanjunxiang-google commented Nov 27, 2023

jbohanon commented Nov 30, 2023 • edited Loading

adisuissa commented Nov 30, 2023

jbohanon commented Nov 30, 2023

yanjunxiang-google commented Nov 30, 2023 • edited Loading

jbohanon commented Dec 1, 2023

yanjunxiang-google commented Nov 22, 2023 •

edited by yanavlasov

Loading

yanjunxiang-google commented Nov 22, 2023 •

edited

Loading

tyxia left a comment •

edited

Loading

tyxia commented Nov 27, 2023 •

edited

Loading

yanjunxiang-google commented Nov 27, 2023 •

edited

Loading

yanjunxiang-google commented Nov 27, 2023 •

edited

Loading

jbohanon commented Nov 30, 2023 •

edited

Loading

yanjunxiang-google commented Nov 30, 2023 •

edited

Loading