Assertion `(parser->current_buffer_len_) == (0)' failed #15102

nitroxis · 2017-08-30T22:06:49Z

Version: v8.1.4
Platform: Linux 4.9.0-2-amd64 deps: update openssl to 1.0.1j #1 SMP Debian 4.9.18-1 (2017-03-30) x86_64 GNU/Linux
Subsystem: node_http_parser.cc

Hi,

I get the following crash from time to time (randomly, it seems):

/usr/local/bin/node[26969]: ../src/node_http_parser.cc:420:static void node::{anonymous}::Parser::Execute(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `(parser->current_buffer_len_) == (0)' failed.
Main process exited, code=killed, status=6/ABRT
 1: node::Abort() [/usr/local/bin/node]
 2: node::Assert(char const* const (*) [4]) [/usr/local/bin/node]
 3: 0x14135d1 [/usr/local/bin/node]
 4: 0xcc6de636c65

I'm not entirely sure where this comes from as there is no JS stack trace, but I believe it happens when processing the response of a HTTP request. This bug has been around for a long time, but it's very inconsistent. It might have already been reported here, at least it seems to be related.

The text was updated successfully, but these errors were encountered:

BridgeAR · 2017-08-30T22:08:45Z

Does the same error occur with 8.4?

nitroxis · 2017-08-30T22:11:00Z

I will try to update node, but it might take weeks until it happens again, unless I find a way to reproduce it.

bnoordhuis · 2017-09-11T09:52:43Z

I'll close this out for now, let us know if you can reproduce with the latest v8.x.

nitroxis · 2017-09-11T14:25:16Z

Coincidentally, it just happened again. Same stack trace, different line:

/usr/local/bin/node[23084]: ../src/node_http_parser.cc:423:static void node::{anonymous}::Parser::Execute(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `(parser->current_buffer_len_) == (0)' failed.
 1: node::Abort() [/usr/local/bin/node]
 2: node::Assert(char const* const (*) [4]) [/usr/local/bin/node]
 3: 0x142b461 [/usr/local/bin/node]
 4: 0x11eabb74aac7
Main process exited, code=killed, status=6/ABRT

I still don't know where exactly in the JS code this happened.

psamim · 2017-09-17T05:28:29Z

It also happened to me, node version 8.4.0 on Archlinux. The 8:54:18 AM api.1 part in the log is from node-foreman.

8:54:18 AM api.1       |  /usr/bin/node[5205]: ../src/node_http_parser.cc:423:static void node::{anonymous}::Parser::Execute(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `(parser->current_buffer_len_) == (0)' failed.
8:54:18 AM api.1       |   1:
8:54:18 AM api.1       |  node::Abort() [/usr/bin/node]
8:54:18 AM api.1       |   2:
8:54:18 AM api.1       |  node::Assert(char const* const (*) [4]) [/usr/bin/node]
8:54:18 AM api.1       |   3: 0x5616b4c073dd [/usr/bin/node]
8:54:18 AM api.1       |   4: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/usr/bin/node]
8:54:18 AM api.1       |   5: 0x5616b442414c [/usr/bin/node]
8:54:18 AM api.1       |   6: 0x5616b4424754 [/usr/bin/node]
8:54:18 AM api.1       |   7: 0x5350f1840dd

addaleax · 2017-09-17T12:50:03Z

@psamim Any chance you could get us a core dump of that crash?

psamim · 2017-09-17T14:18:34Z

I do not know how.

…

On Sun, Sep 17, 2017 at 5:20 PM Anna Henningsen ***@***.***> wrote: @psamim <https://github.com/psamim> Any chance you could get us a core dump of that crash? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15102 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AByDh44MUSI-Ma64CJV0WGCVtBCCLETMks5sjRWggaJpZM4PICpG> .

SINHASantos · 2017-09-18T09:41:21Z

linux node v8.4.0

/usr/local/bin/node[30809]: ../src/node_http_parser.cc:423:static void node::{a nonymous}::Parser::Execute(const v8::FunctionCallbackInfov8::Value&): Asserti on `(parser->current_buffer_len_) == (0)' failed.
1: node::Abort() [node]
2: node::Assert(char const* const () [4]) [node]
3: 0x13c5bcd [node]
4: v8::internal::FunctionCallbackArguments::Call(void ()(v8::FunctionCallback Infov8::Value const&)) [node]
5: 0xc3ac4a [node]
6: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::intern al::Isolate*) [node]
7: 0x39c5acd040dd
Aborted

solution:

install following module

npm install http-parser-js

and add the line below in your code

process.binding('http_parser').HTTPParser = require('http-parser-js').HTTPParser;

and it solves the problem

cjihrig · 2017-09-18T13:05:04Z

I do not know how.

@psamim start Node with the --abort-on-uncaught-exception flag. Before starting node, also run ulimit -c unlimited.

psamim · 2017-09-19T12:31:59Z

@cjihrig I did and the same thing happened:

3:56:58 PM api.1       |  /usr/bin/node[22446]: ../src/node_http_parser.cc:423:static void node::{anonymous}::Parser::Execute(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `(parser->current_buffer_len_) == (0)' failed.
3:56:58 PM api.1       |   1:
3:56:58 PM api.1       |  node::Abort()
3:56:58 PM api.1       |   [/usr/bin/node]
3:56:58 PM api.1       |   2:
3:56:58 PM api.1       |  node::Assert(char const* const (*) [4])
3:56:58 PM api.1       |   [/usr/bin/node]
3:56:58 PM api.1       |   3:
3:56:58 PM api.1       |  0x5616f1cbd3dd [/usr/bin/node]
3:56:58 PM api.1       |   4: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/usr/bin/node]
3:56:58 PM api.1       |   5: 0x5616f14da14c [/usr/bin/node]
3:56:58 PM api.1       |   6:
3:56:58 PM api.1       |  0x5616f14da754 [/usr/bin/node]
3:56:58 PM api.1       |   7: 0x3f5c6b9840dd

addaleax · 2017-09-19T12:35:03Z

@psamim Do you find a file name core or similar where you were running your node program?

psamim · 2017-09-19T14:06:59Z

There are 125 files named `core` but none are in the root directory where I run the program.

…

On Tue, Sep 19, 2017 at 5:06 PM Anna Henningsen ***@***.***> wrote: @psamim <https://github.com/psamim> Do you find a file name core or similar where you were running your node program? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15102 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AByDhzjLCd0_LaKLisnl7Jse3h9y7oOZks5sj7VNgaJpZM4PICpG> .

addaleax · 2017-09-19T16:55:11Z

@psamim does any of the creation times of those files match the time of the crash as per the logs? that would be a good indicator that it’s what we’re looking for :)

psamim · 2017-09-26T16:41:14Z

The problem seems resolved for me, in version 8.5.0.

bnoordhuis · 2018-05-28T19:28:08Z

The problem seems resolved for me, in version 8.5.0.

Happy to hear that. I'll close out the issue.

himanshu-singh1995 · 2019-07-08T12:25:52Z

I got the following error in node v10.16.0 while uploading a 1GB file on S3 using putObject method provided by the aws-sdk. By the uploading a file size of 1GB to S3 using putObject of aws-sdk, I was able to reproduce this error every time.

Note: Upload string(not Buffer) to S3.

node[10695]: ../src/node_http_parser.cc:442:static void node::{anonymous}::Parser::Finish(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `parser->current_buffer_.IsEmpty()' failed.
 1: 0x8f9d10 node::Abort() [node]
 2: 0x8f9de5  [node]
 3: 0x9638b2  [node]
 4: 0xb8dfef  [node]
 5: 0xb8eb59 v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) [node]
 6: 0x154a4c55be1d 
Aborted (core dumped)

Code to reproduce this error

const fs = require('fs');
const AWS = require('aws-sdk');
AWS.config.update(/* creds */);

let data = null
fs.readFile('filename','utf8', (err, _data) => data = _data)  // this file should be >= 1GB
const params = {
      ACL: 'public-read',
      Body: data,
      Bucket: '<bucket>',
      Key: '<key>',
      ContentType: 'text/plain',
    };
const s3 = new AWS.S3({ apiVersion: '2006-03-01' });
s3.putObject(params).promise().then(console.log, console.log);

jsohan · 2020-07-06T15:28:38Z

@himanshu-singh1995 I think I am facing the same issue. Did you find a fix or work around for this. Also was it any file 1GB and bigger?

addaleax · 2020-07-07T17:13:53Z

@himanshu-singh1995 @jsohan Can you tell whether you are affected by this on Node v12.x or v14.x as well?

jsohan · 2020-07-07T17:41:47Z

@addaleax This effects me on Node v12. Not sure what the root cause it but this error is being thrown from s3.putObject() function when dealing with large strings

addaleax · 2020-07-07T17:47:27Z

@jsohan Is there a chance you could share a core dump (+ your exact Node.js version, assuming you’re using the binaries from https://nodejs.org/, or if not, the Node.js executable you are using)? It would be good to know what the JS stack trace is here, but unfortunately the crash output here does not provide that…

jsohan · 2020-07-07T19:11:23Z

@addaleax Not to sure what a core Dump is but I'm running node v12.14.0 locally . Error was first thrown in production but I'm not sure the exact node version ran there. When I reproduce the bug locally I run the following code:

function getData(){
    return new Promise(resolve => {
      fs.readFile('temp.txt','utf8', (err, _data) => {
        return resolve(_data);
      })  // this file should be >= 1GB

    })
  }

  async function tempFunction(){
    console.log("tempFunction called ***************************************");
    let data = await getData(); //gets data
    var params = {
      Bucket: s3Config.bucket,
      Key: "temp3",
      Body: data
    };
    console.log("passing: temp | FileSize: " + Buffer.byteLength(data, 'utf8') + " bytes");
    await aws3.putObject(params).promise() //uploads data
      .then(async (data) => {
        console.log("done uploading");
      })
      .catch(err => {
        console.log("error was thrown in tempFunction")
        console.log(err);
      })
  }

And I get the following console output:

Skipping unexpected line on STDOUT for shard shardId-000000000000: tempFunction called *************************************** 
Skipping unexpected line on STDOUT for shard shardId-000000000000: passing: temp | FileSize: 1058393952 bytes 
Received error line from subprocess [C:\Program Files\nodejs\node.exe[21196]: c:\ws\src\node_http_parser_impl.h:488: Assertion `parser->current_buffer_.IsEmpty()' failed.] for shard shardId-000000000000 
C:\Program Files\nodejs\node.exe[21196]: c:\ws\src\node_http_parser_impl.h:488: Assertion `parser->current_buffer_.IsEmpty()' failed.
Received error line from subprocess [ 1: 00007FF6BC67124F napi_wrap+124431] for shard shardId-000000000000 
 1: 00007FF6BC67124F napi_wrap+124431
 2: 00007FF6BC612A06 v8::base::CPU::has_sse+34502
 3: 00007FF6BC612D23 v8::base::CPU::has_sse+35299
Received error line from subprocess [ 2: 00007FF6BC612A06 v8::base::CPU::has_sse+34502] for shard shardId-000000000000 
Received error line from subprocess [ 3: 00007FF6BC612D23 v8::base::CPU::has_sse+35299] for shard shardId-000000000000 
Received error line from subprocess [ 4: 00007FF6BC60079E v8::internal::Scope::locals+181358] for shard shardId-000000000000 
 4: 00007FF6BC60079E v8::internal::Scope::locals+181358
Received error line from subprocess [ 5: 00007FF6BCDCC4E0 v8::internal::Builtins::builtin_handle+325536] for shard shardId-000000000000 
 5: 00007FF6BCDCC4E0 v8::internal::Builtins::builtin_handle+325536
 6: 00007FF6BCDCBA27 v8::internal::Builtins::builtin_handle+322791
Received error line from subprocess [ 6: 00007FF6BCDCBA27 v8::internal::Builtins::builtin_handle+322791] for shard shardId-000000000000 
Received error line from subprocess [ 7: 00007FF6BCDCBD68 v8::internal::Builtins::builtin_handle+323624] for shard shardId-000000000000

Sorry if the formatting is little weird. This is being thrown from the record processor function of aws-kcl library, getting data from a Kinesis stream
The error is a little different locally but in production it's more like it is in @himanshu-singh1995 post

addaleax · 2020-07-07T19:21:07Z

@jsohan Sorry, didn’t know you’re running Windows… in that case, yeah, core dumps or C++ stack traces won’t be available/useful.

Can you try adding this to your code (before you make the request), and run Node.js with --http-parser=legacy?

const { HTTPParser } = process.binding('http_parser');
const origFinish = HTTPParser.prototype.finish;
HTTPParser.prototype.finish = function(...args) {
  console.trace('finish called', args);
  return origFinish.call(this, ...args);
};

that should give us the JS stack trace, which is the most useful thing we can get here.

jsohan · 2020-07-07T19:55:26Z

@addaleax here's what I got

Skipping unexpected line on STDOUT for shard shardId-000000000000: tempFunction called *************************************** 
Skipping unexpected line on STDOUT for shard shardId-000000000000: passing: temp | FileSize: 1058393952 bytes 
Received error line from subprocess [Trace: finish called []] for shard shardId-000000000000 
Trace: finish called []
    at HTTPParser.finish (c:\Users\jsohan\Documents\gitRepo\abacus-nodejs\archiver.js:16:11)
    at Socket.socketErrorListener (_http_client.js:414:12)
Received error line from subprocess [    at HTTPParser.finish (c:\Users\jsohan\Documents\gitRepo\abacus-nodejs\archiver.js:16:11)] for shard shardId-000000000000 
Received error line from subprocess [    at Socket.socketErrorListener (_http_client.js:414:12)] for shard shardId-000000000000 
Received error line from subprocess [    at Socket.emit (events.js:210:5)] for shard shardId-000000000000 
Received error line from subprocess [    at Socket.EventEmitter.emit (domain.js:475:20)] for shard shardId-000000000000 
Received error line from subprocess [    at errorOrDestroy (internal/streams/destroy.js:108:12)] for shard shardId-000000000000 
    at Socket.emit (events.js:210:5)
    at Socket.EventEmitter.emit (domain.js:475:20)
    at errorOrDestroy (internal/streams/destroy.js:108:12)
    at onwriteError (_stream_writable.js:446:5)
Received error line from subprocess [    at onwriteError (_stream_writable.js:446:5)] for shard shardId-000000000000 
Received error line from subprocess [    at onwrite (_stream_writable.js:473:5)] for shard shardId-000000000000 
    at onwrite (_stream_writable.js:473:5)
    at internal/streams/destroy.js:50:7
Received error line from subprocess [    at internal/streams/destroy.js:50:7] for shard shardId-000000000000 
Received error line from subprocess [    at Socket._destroy (net.js:664:5)] for shard shardId-000000000000 
    at Socket._destroy (net.js:664:5)
    at Socket.destroy (internal/streams/destroy.js:38:8)
Received error line from subprocess [    at Socket.destroy (internal/streams/destroy.js:38:8)] for shard shardId-000000000000 
Received error line from subprocess [C:\Program Files\nodejs\node.exe[17300]: c:\ws\src\node_http_parser_impl.h:488: Assertion `parser->current_buffer_.IsEmpty()' failed.] for shard shardId-000000000000 
C:\Program Files\nodejs\node.exe[17300]: c:\ws\src\node_http_parser_impl.h:488: Assertion `parser->current_buffer_.IsEmpty()' failed.
Received error line from subprocess [ 1: 00007FF6BC67124F napi_wrap+124431] for shard shardId-000000000000 
 1: 00007FF6BC67124F napi_wrap+124431
 2: 00007FF6BC612A06 v8::base::CPU::has_sse+34502
Received error line from subprocess [ 2: 00007FF6BC612A06 v8::base::CPU::has_sse+34502] for shard shardId-000000000000 
Received error line from subprocess [ 3: 00007FF6BC612D23 v8::base::CPU::has_sse+35299] for shard shardId-000000000000 
 3: 00007FF6BC612D23 v8::base::CPU::has_sse+35299
 4: 00007FF6BC5FC9BE v8::internal::Scope::locals+165518
Received error line from subprocess [ 4: 00007FF6BC5FC9BE v8::internal::Scope::locals+165518] for shard shardId-000000000000 
Received error line from subprocess [ 5: 00007FF6BCDCC4E0 v8::internal::Builtins::builtin_handle+325536] for shard shardId-000000000000 
 5: 00007FF6BCDCC4E0 v8::internal::Builtins::builtin_handle+325536
 6: 00007FF6BCDCBA27 v8::internal::Builtins::builtin_handle+322791
Received error line from subprocess [ 6: 00007FF6BCDCBA27 v8::internal::Builtins::builtin_handle+322791] for shard shardId-000000000000 
Received error line from subprocess [ 7: 00007FF6BCDCBD68 v8::internal::Builtins::builtin_handle+323624] for shard shardId-000000000000 
 7: 00007FF6BCDCBD68 v8::internal::Builtins::builtin_handle+323624
Received error line from subprocess [ 8: 00007FF6BCDCBB6E v8::internal::Builtins::builtin_handle+323118] for shard shardId-000000000000 
Received error line from subprocess [ 9: 00007FF6BD24414D v8::internal::SetupIsolateDelegate::SetupHeap+568205] for shard shardId-000000000000 
 8: 00007FF6BCDCBB6E v8::internal::Builtins::builtin_handle+323118
 9: 00007FF6BD24414D v8::internal::SetupIsolateDelegate::SetupHeap+568205
10: 00007FF6BD1C3BDC v8::internal::SetupIsolateDelegate::SetupHeap+42524

addaleax · 2020-07-07T19:58:58Z

@jsohan Thanks, that seems quite helpful already – can you expand the stack trace a bit more, e.g. by adding --stack-trace-limit=100? That should be even more helpful here. (Thanks for being patient with figuring this out!)

jsohan · 2020-07-07T20:10:29Z

@addaleax Would be awesome to find out what's going wrong here

addaleax · 2020-07-07T20:39:33Z

@jsohan Awesome, thank you! I managed to put together a fully self-contained reproduction based on this :)

The good news is that this already appears to be fixed (at least to some degree) on Node v14.x, so it might just be a matter of backporting the right fix + adding a test.

jsohan · 2020-07-07T20:46:46Z

Awesome. Any idea on how I would go about this? @addaleax

addaleax · 2020-07-07T20:49:13Z

For a workaround, you might be able to wrap the self.writeBody(stream, httpRequest); line in aws-sdk/lib/http/node.js:102 as setImmediate(() => self.writeBody(stream, httpRequest));, I think that would fix this for you. (Unfortunately, there probably isn’t any way to fix this in your code without monkey-patching either the AWS library or Node’s HTTP internals…)

addaleax · 2020-07-07T21:26:48Z

Looks like the fix was f663b31, which was semver-major, so I’ll open a separate fix instead of backporting that

Since the tests only crash on v12.x, this commit adds separate regression tests. Refs: nodejs#15102

Fix a crash that occurs when `parser.finish()` is called during `parser.execute()`. In this particular case, this happened because a 100 continue response is a place in which `.end()` can be called which can in turn lead to a write error, which is emitted synchronously, thus inside the outer `parser.execute()` call. Resolve that by delaying the `parser.finish()` call until after the `parser.execute()` call is done. This only affects v12.x, because on later versions, errors are not emitted synchronously. Fixes: nodejs#15102

Since the tests only crash on v12.x, this commit adds separate regression tests. Refs: nodejs#15102

Since the tests only crash on v12.x, this commit adds separate regression tests. Refs: #15102 PR-URL: #34250 Refs: #34016 Reviewed-By: Luigi Pinca <[email protected]>

Since the tests only crash on v12.x, this commit adds separate regression tests. PR-URL: #34251 Refs: #15102 Refs: #34016 Reviewed-By: Fedor Indutny <[email protected]> Reviewed-By: Gerhard Stöbich <[email protected]>

mscdex added c++ Issues and PRs that require attention from people who are familiar with C++. http Issues or PRs related to the http subsystem. labels Aug 30, 2017

bnoordhuis closed this as completed Sep 11, 2017

BridgeAR reopened this Sep 17, 2017

bnoordhuis closed this as completed May 28, 2018

addaleax reopened this Jul 7, 2020

addaleax added a commit to addaleax/node that referenced this issue Jul 7, 2020

test: add regression tests for HTTP parser crash

59408fb

Since the tests only crash on v12.x, this commit adds separate regression tests. Refs: nodejs#15102

addaleax added a commit to addaleax/node that referenced this issue Jul 7, 2020

test: add regression tests for HTTP parser crash

11406b9

Since the tests only crash on v12.x, this commit adds separate regression tests. Refs: nodejs#15102

This was referenced Jul 7, 2020

test: add regression tests for HTTP parser crash #34250

Closed

[v12.x] http: fix crash for sync write errors during header parsing #34251

Closed

addaleax added a commit that referenced this issue Jul 14, 2020

test: add regression tests for HTTP parser crash

18667ac

Since the tests only crash on v12.x, this commit adds separate regression tests. Refs: #15102 PR-URL: #34250 Refs: #34016 Reviewed-By: Luigi Pinca <[email protected]>

MylesBorins pushed a commit that referenced this issue Jul 14, 2020

test: add regression tests for HTTP parser crash

a70a5a5

Since the tests only crash on v12.x, this commit adds separate regression tests. Refs: #15102 PR-URL: #34250 Refs: #34016 Reviewed-By: Luigi Pinca <[email protected]>

MylesBorins pushed a commit that referenced this issue Jul 16, 2020

test: add regression tests for HTTP parser crash

3ee8f53

Since the tests only crash on v12.x, this commit adds separate regression tests. Refs: #15102 PR-URL: #34250 Refs: #34016 Reviewed-By: Luigi Pinca <[email protected]>

cjihrig pushed a commit that referenced this issue Jul 23, 2020

test: add regression tests for HTTP parser crash

24df1c0

Since the tests only crash on v12.x, this commit adds separate regression tests. Refs: #15102 PR-URL: #34250 Refs: #34016 Reviewed-By: Luigi Pinca <[email protected]>

santigimeno closed this as completed Oct 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Assertion `(parser->current_buffer_len_) == (0)' failed #15102

Assertion `(parser->current_buffer_len_) == (0)' failed #15102

nitroxis commented Aug 30, 2017

BridgeAR commented Aug 30, 2017

nitroxis commented Aug 30, 2017

bnoordhuis commented Sep 11, 2017

nitroxis commented Sep 11, 2017

psamim commented Sep 17, 2017

addaleax commented Sep 17, 2017

psamim commented Sep 17, 2017 via email

SINHASantos commented Sep 18, 2017 •

edited

Loading

cjihrig commented Sep 18, 2017

psamim commented Sep 19, 2017

addaleax commented Sep 19, 2017

psamim commented Sep 19, 2017 via email

addaleax commented Sep 19, 2017 •

edited

Loading

psamim commented Sep 26, 2017

bnoordhuis commented May 28, 2018

himanshu-singh1995 commented Jul 8, 2019 •

edited

Loading

jsohan commented Jul 6, 2020

addaleax commented Jul 7, 2020

jsohan commented Jul 7, 2020

addaleax commented Jul 7, 2020

jsohan commented Jul 7, 2020 •

edited

Loading

addaleax commented Jul 7, 2020

jsohan commented Jul 7, 2020

addaleax commented Jul 7, 2020

jsohan commented Jul 7, 2020

addaleax commented Jul 7, 2020

jsohan commented Jul 7, 2020

addaleax commented Jul 7, 2020

addaleax commented Jul 7, 2020

Assertion `(parser->current_buffer_len_) == (0)' failed #15102

Assertion `(parser->current_buffer_len_) == (0)' failed #15102

Comments

nitroxis commented Aug 30, 2017

BridgeAR commented Aug 30, 2017

nitroxis commented Aug 30, 2017

bnoordhuis commented Sep 11, 2017

nitroxis commented Sep 11, 2017

psamim commented Sep 17, 2017

addaleax commented Sep 17, 2017

psamim commented Sep 17, 2017 via email

SINHASantos commented Sep 18, 2017 • edited Loading

cjihrig commented Sep 18, 2017

psamim commented Sep 19, 2017

addaleax commented Sep 19, 2017

psamim commented Sep 19, 2017 via email

addaleax commented Sep 19, 2017 • edited Loading

psamim commented Sep 26, 2017

bnoordhuis commented May 28, 2018

himanshu-singh1995 commented Jul 8, 2019 • edited Loading

jsohan commented Jul 6, 2020

addaleax commented Jul 7, 2020

jsohan commented Jul 7, 2020

addaleax commented Jul 7, 2020

jsohan commented Jul 7, 2020 • edited Loading

addaleax commented Jul 7, 2020

jsohan commented Jul 7, 2020

addaleax commented Jul 7, 2020

jsohan commented Jul 7, 2020

addaleax commented Jul 7, 2020

jsohan commented Jul 7, 2020

addaleax commented Jul 7, 2020

addaleax commented Jul 7, 2020

SINHASantos commented Sep 18, 2017 •

edited

Loading

addaleax commented Sep 19, 2017 •

edited

Loading

himanshu-singh1995 commented Jul 8, 2019 •

edited

Loading

jsohan commented Jul 7, 2020 •

edited

Loading