Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

undefined behavior in quic integration test #34353

Closed
kyessenov opened this issue May 24, 2024 · 4 comments · Fixed by #34697
Closed

undefined behavior in quic integration test #34353

kyessenov opened this issue May 24, 2024 · 4 comments · Fixed by #34697
Assignees
@danzh2010
Labels
area/quic area/test flakes bug

Comments

@kyessenov
Copy link
Contributor

Log https://dev.azure.com/cncf/envoy/_build/results?buildId=171258&view=logs&jobId=1439b9f7-a348-5b50-b5fe-ea612ea91241&j=1439b9f7-a348-5b50-b5fe-ea612ea91241&t=1002ac43-da84-5fae-70b2-98833b702d09&s=4f7d954b-a765-565f-91a2-c04870dab43f

[ RUN      ] QuicHttpIntegrationTests/QuicHttpIntegrationTest.MultipleQuicConnectionsNoBPF/IPv4
source/common/http/http1/codec_impl.cc:1067:9: runtime error: member call on address 0x61c00003d080 which does not point to an object of type 'Envoy::Http::Http1::ServerConnectionImpl'
0x61c00003d080: note: object has invalid vptr
 19 01 00 00  73 67 12 00 00 00 00 00  be be be be be be be be  78 25 a4 0f 00 00 00 00  88 5e f7 0f
              ^~~~~~~~~~~~~~~~~~~~~~~
              invalid vptr
    #0 0xaa84996 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5&, Envoy::Buffer::OwnedBufferFragmentImpl const*>(Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5&, Envoy::Buffer::OwnedBufferFragmentImpl const*&&) /proc/self/cwd/source/common/http/http1/codec_impl.cc:1067:9
    #1 0xaa8467a in std::__1::__function::__func<Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5, std::__1::allocator<Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5>, void (Envoy::Buffer::OwnedBufferFragmentImpl const*)>::operator()(Envoy::Buffer::OwnedBufferFragmentImpl const*&&) /opt/llvm/bin/../include/c++/v1/__functional/function.h:180:16
    #2 0xaa91e20 in std::__1::__function::__value_func<void (Envoy::Buffer::OwnedBufferFragmentImpl const*)>::operator()(Envoy::Buffer::OwnedBufferFragmentImpl const*&&) const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
    #3 0xaa919dd in Envoy::Buffer::OwnedBufferFragmentImpl::done() /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
    #4 0xe6968b8 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::Buffer::Slice::Slice(Envoy::Buffer::BufferFragment&)::'lambda'()&>(Envoy::Buffer::Slice::Slice(Envoy::Buffer::BufferFragment&)::'lambda'()&) /proc/self/cwd/./source/common/buffer/buffer_impl.h:95:42


SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior source/common/http/http1/codec_impl.cc:1067:9 in
@kyessenov kyessenov added bug triage Issue requires triage labels May 24, 2024
@zuercher zuercher added area/quic area/test flakes and removed triage Issue requires triage labels May 28, 2024
@zuercher
Copy link
Member

cc @danzh2010

@danzh2010
Copy link
Contributor

UBSAN alerts in Envoy::Http::Http1::ServerConnectionImpl, instead of QUIC objects. I'm not an expert in that code stack though.

@danzh2010 danzh2010 removed their assignment Jun 5, 2024
@danzh2010
Copy link
Contributor

@alyssawilk I don't think this is related to HTTP/3. The call stack has Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl.

@alyssawilk
Copy link
Contributor

Sure but AFIK it's only showing up in an H3 test, so it may be a lifetime bug in that particular test

This was referenced Jun 11, 2024
Open
Merged
alyssawilk pushed a commit that referenced this issue Jun 14, 2024
@danzh2010 @danzh1989
quic: fix ASAN in QuicHttpIntegrationTest.QuicMultipleQuicConnections…
a61a972 
…NoBPF (#34697)

Commit Message: switch to use autonomous upstream in testMultipleQuicConnections(). The ASAN complains about the multiple resets of fake_upstream_connection_ during the tests which races with the FakeUpstream's network connection's I/O events in the upstream thread. This test helper function doesn't need to use controlled test upstream, so using autonomous upstream should avoid the race.

This potentially fix #34353 which I wasn't able to reproduce locally under ASAN. This test has to run with --jobs=1 if we set --runs_under_test to be larger than 1. 1000 runs already took more than 1h to finish.

Risk Level: low, test onlhy
Testing: existing test passes under ASAN
Docs Changes: N/A
Release Notes: N/A
Platform Specific Features: N/A
Fixes #34353

Signed-off-by: Dan Zhang <[email protected]>
Co-authored-by: Dan Zhang <[email protected]>
Nealsoni00 pushed a commit to Nealsoni00/envoy that referenced this issue Jun 18, 2024
@danzh2010 @danzh1989 @Nealsoni00
quic: fix ASAN in QuicHttpIntegrationTest.QuicMultipleQuicConnections…
32d5c43 
…NoBPF (envoyproxy#34697)

Commit Message: switch to use autonomous upstream in testMultipleQuicConnections(). The ASAN complains about the multiple resets of fake_upstream_connection_ during the tests which races with the FakeUpstream's network connection's I/O events in the upstream thread. This test helper function doesn't need to use controlled test upstream, so using autonomous upstream should avoid the race.

This potentially fix envoyproxy#34353 which I wasn't able to reproduce locally under ASAN. This test has to run with --jobs=1 if we set --runs_under_test to be larger than 1. 1000 runs already took more than 1h to finish.

Risk Level: low, test onlhy
Testing: existing test passes under ASAN
Docs Changes: N/A
Release Notes: N/A
Platform Specific Features: N/A
Fixes envoyproxy#34353

Signed-off-by: Dan Zhang <[email protected]>
Co-authored-by: Dan Zhang <[email protected]>
Signed-off-by: Neal Soni <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danzh2010 danzh2010

Labels
area/quic area/test flakes bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

quic: fix ASAN in QuicHttpIntegrationTest.QuicMultipleQuicConnectionsNoBPF danzh2010/envoy
4 participants
@zuercher @kyessenov @alyssawilk @danzh2010