Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ASan hit on background threads broken with "TypeError: Cannot read property 'length' of undefined" #13205

Closed
emaxx-google opened this issue Jan 8, 2021 · 7 comments · Fixed by #14611

Comments

@emaxx-google
Copy link

emaxx-google commented Jan 8, 2021

If Asan failure happens on the background thread, "TypeError: Cannot read property 'length' of undefined" is generated instead of the actual ASan output.

Example program:

#include <iostream>
#include <thread>
int main() {
  std::thread([]() {
    int* p = new int(1);
    delete p;
    std::cout << *p;
  }).detach();
}

Build flags:

--std=c++11 -pthread -s INITIAL_MEMORY=67108864 -s PTHREAD_POOL_SIZE=2 -fsanitize=address

The full console output when running in the browser:

=================================================================
==42==ERROR: AddressSanitizer: heap-use-after-free on address 0x02e00f70 at pc 0x00004403 bp 0x031f4780 sp 0x031f478c
READ of size 4 at 0x02e00f70 thread T1
Pthread 0x3203ec0 exited.
worker.js onmessage() captured an uncaught exception: TypeError: Cannot read property 'length' of undefined
TypeError: Cannot read property 'length' of undefined
    at WasmOffsetConverter.getIndex (a.out.js:1427)
    at WasmOffsetConverter.getName (a.out.js:1445)
    at _emscripten_pc_get_function (a.out.js:5136)
    at a.out.wasm:wasm-function[2305]:0x4aa65
    at a.out.wasm:wasm-function[2300]:0x4a79c
    at a.out.wasm:wasm-function[2523]:0x53863
    at a.out.wasm:wasm-function[2112]:0x41123
    at a.out.wasm:wasm-function[2212]:0x456b7
    at a.out.wasm:wasm-function[2210]:0x45450
    at a.out.wasm:wasm-function[2224]:0x46379

Emscripten version: ToT (since ASan+pthreads is broken in older releases - see #13183).

@sbc100
Copy link
Collaborator

sbc100 commented Jan 12, 2021

This works for me with ToT... at least it generates an asan warning:

$ node --experimental-wasm-bulk-memory  --experimental-wasm-threads a.out.js 
=================================================================
==42==ERROR: AddressSanitizer: heap-use-after-free on address 0x03000f70 at pc 0x00004c72 bp 0x033f8780 sp 0x033f878c
READ of size 4 at 0x03000f70 thread T1
    #0 0x4c72 in main::$_0::operator()() const+0x4c72 (a.out.wasm+0x4c72)

0x03000f70 is located 0 bytes inside of 4-byte region [0x03000f70,0x03000f74)
freed by thread T1 here:
    #0 0x48773 in operator delete(void*)+0x48773 (a.out.wasm+0x48773)
    #1 0x4bcd in main::$_0::operator()() const+0x4bcd (a.out.wasm+0x4bcd)
    #2 0x4a05 in decltype(std::__2::forward<main::$_0>(fp)()) std::__2::__invoke<main::$_0>(main::$_0&&)+0x4a05 (a.out.wasm+0x4a05)
    #3 0x369d in void std::__2::__thread_execute<std::__2::unique_ptr<std::__2::__thread_struct, std::__2::default_delete<std::__2::__thread_struct> >, main::$_0>(std::__2::tuple<std::__2::unique_ptr<std::__2::__thread_struct, std::__2::default_delete<std::__2::__thread_struct> >, main::$_0>&, std::__2::__tuple_indices<>)+0x369d (a.out.wasm+0x369d)
Pthread aborting at Error
    at abort (eval at globalEval (/home/sbc/dev/wasm/emscripten/a.out.worker.js:228:10), <anonymous>:1253:69)
    at abortOnCannotGrowMemory (eval at globalEval (/home/sbc/dev/wasm/emscripten/a.out.worker.js:228:10), <anonymous>:5181:2)
    at _emscripten_resize_heap (eval at globalEval (/home/sbc/dev/wasm/emscripten/a.out.worker.js:228:10), <anonymous>:5186:2)
    at sbrk (wasm-function[1933]:0x36f85)
    at dlmalloc (wasm-function[1927]:0x34e86)
    at internal_memalign (wasm-function[1931]:0x366ff)
    at dlmemalign (wasm-function[1930]:0x36685)
    at eval (eval at globalEval (/home/sbc/dev/wasm/emscripten/a.out.worker.js:228:10), <anonymous>:1292:20)
    at syscallMmap2 (eval at globalEval (/home/sbc/dev/wasm/emscripten/a.out.worker.js:228:10), <anonymous>:4818:9)
    at eval (eval at globalEval (/home/sbc/dev/wasm/emscripten/a.out.worker.js:228:10), <anonymous>:4843:10)

But that looks like a separate issue. Can you confirm that the latest emscripten ToT works for you too? (and close this issue if so).

@emaxx-google
Copy link
Author

Works for me on "tot" too, so it's been fixed by one of the recent pthreads/asan fixes. Thanks!

@emaxx-google
Copy link
Author

Reopening this - I believe it still reproes, but only when running it in the browser (and not in Node). That was the crucial part that was mentioned in the initial comment and forgotten later; apologies for missing it when closing the issue.

I've double-checked that the issue is present on a fresh ToT release (release-4d185e1410f1d5f159498fa776279df88d124f54, emscripten commit 3307679).

A fuller stack trace (collected by adding -O0 -g) is:

TypeError: Cannot read property 'length' of undefined
    at WasmOffsetConverter.getIndex (a.out.js:1443)
    at WasmOffsetConverter.getName (a.out.js:1461)
    at _emscripten_pc_get_function (a.out.js:5119)
    at __sanitizer::EmscriptenSymbolizerTool::SymbolizePC(unsigned long, __sanitizer::SymbolizedStack*) (a.out.wasm:wasm-function[2305]:0x4ce2c)
    at __sanitizer::Symbolizer::SymbolizePC(unsigned long) (a.out.wasm:wasm-function[2300]:0x4cb45)
    at __sanitizer::StackTrace::Print() const (a.out.wasm:wasm-function[2523]:0x56180)
    at __asan::ErrorGeneric::Print() (a.out.wasm:wasm-function[2112]:0x42e10)
    at __asan::ErrorDescription::Print() (a.out.wasm:wasm-function[2212]:0x4762a)
    at __asan::ScopedInErrorReport::~ScopedInErrorReport() (a.out.wasm:wasm-function[2210]:0x473bc)
    at __asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) (a.out.wasm:wasm-function[2224]:0x48351)

@emaxx-google emaxx-google reopened this Jan 18, 2021
@emaxx-google
Copy link
Author

Hello, is there any workaround for this issue? (Besides running the program under Node, since it's not possible for the real-world program I'm trying to debug, due to it relying on the Extension APIs.)

@sbc100
Copy link
Collaborator

sbc100 commented Feb 6, 2021

I'm not aware of any fix this yet

@emaxx-google
Copy link
Author

I'm seeing a similar issue when running nontrivial programs under Node.js as well:

================================================================= 
==42==ERROR: AddressSanitizer: heap-use-after-free on address 0x054b3194 at pc 0x001aa0ec bp 0x07f60730 sp 0x07f6073c 
READ of size 4 at 0x054b3194 thread T2 
pthread sent an error! undefined:undefined: Cannot read property 'getName' of undefined

(no minified repro for this)

@tocubed
Copy link

tocubed commented May 12, 2021

In my case, it seems that wasmOffsetConverter is not being initialized prior to the worker thread receiving it via. postMessage. In the code below the second promise gets called and sends over an undefined value in place of wasmOffsetConverter. I was able to workaround this by chaining the first promise to the second.

    var result = WebAssembly.instantiateStreaming(response, info);
    Promise.all([ response.clone().arrayBuffer(), result ]).then(function(results) {
     wasmOffsetConverter = new WasmOffsetConverter(new Uint8Array(results[0]), results[1].module);
     if (!ENVIRONMENT_IS_PTHREAD) {
      removeRunDependency("offset-converter");
     }
    }, function(reason) {
     err("failed to initialize offset-converter: " + reason);
    });
    return result.then(receiveInstantiatedSource, function(reason) {
     err("wasm streaming compile failed: " + reason);
     err("falling back to ArrayBuffer instantiation");
     return instantiateArrayBuffer(receiveInstantiatedSource);
    });

kripken added a commit that referenced this issue Jul 9, 2021
If ASan wants to show a stack trace on a pthread (like the trace to get to
an allocation that was later used after free), then it needs access to the
WasmOffsetConverter. We had a race there, where it could postMessage
the WasmOffsetConverter before that object was actually filled with the
data it needs (function offsets). See details in comments.

Fixes #13205
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants