Flaky parallel/test-trace-events-fs-sync #21038

mmarchini · 2018-05-30T17:53:02Z

Test failing occasionally on FreeBSD with different assertion failures:

https://ci.nodejs.org/job/node-test-commit-freebsd/18147/ (#20962)

assert.js:80
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: fs.sync.access:
    at Object.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-trace-events-fs-sync.js:131:10)
    at Module._compile (internal/modules/cjs/loader.js:702:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
    at Module.load (internal/modules/cjs/loader.js:612:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
    at Function.Module._load (internal/modules/cjs/loader.js:543:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:744:10)
    at startup (internal/bootstrap/node.js:239:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:573:3)

https://ci.nodejs.org/job/node-test-commit-freebsd/18103/ (#21003)

assert.js:80
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: fs.sync.ftruncate:
    at Object.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-trace-events-fs-sync.js:131:10)
    at Module._compile (internal/modules/cjs/loader.js:702:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
    at Module.load (internal/modules/cjs/loader.js:612:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
    at Function.Module._load (internal/modules/cjs/loader.js:543:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:744:10)
    at startup (internal/bootstrap/node.js:239:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:573:3)

https://ci.nodejs.org/job/node-test-commit-freebsd/18143/ (#21032)

assert.js:80
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: fs.sync.readlink:
    at Object.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-trace-events-fs-sync.js:131:10)
    at Module._compile (internal/modules/cjs/loader.js:702:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
    at Module.load (internal/modules/cjs/loader.js:612:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
    at Function.Module._load (internal/modules/cjs/loader.js:543:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:744:10)
    at startup (internal/bootstrap/node.js:239:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:573:3)

The text was updated successfully, but these errors were encountered:

test-trace-events-fs-sync has been failing ocasinally on FreeBSD. While we don't have a fix, it should be marked as flaky. Ref: nodejs#21038

Trott · 2018-05-31T03:10:21Z

Seems like this assertion should print the entire proc object rather than just proc.status. This way we could see proc.error, proc.signal, proc.stderr and other things that might be useful to see when the test fails.

Trott · 2018-06-04T08:23:44Z

Not just FreeBSD. Here it is failing on LinuxONE:

https://ci.nodejs.org/job/node-test-commit-linuxone/1863/nodes=rhel72-s390x/console

03:04:07 not ok 1843 parallel/test-trace-events-fs-sync
03:04:07   ---
03:04:07   duration_ms: 2.122
03:04:07   severity: fail
03:04:07   exitcode: 1
03:04:07   stack: |-
03:04:07     assert.js:80
03:04:07       throw new AssertionError(obj);
03:04:07       ^
03:04:07     
03:04:07     AssertionError [ERR_ASSERTION]: fs.sync.rmdir: 
03:04:07         at Object.<anonymous> (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/parallel/test-trace-events-fs-sync.js:131:10)
03:04:07         at Module._compile (internal/modules/cjs/loader.js:702:30)
03:04:07         at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
03:04:07         at Module.load (internal/modules/cjs/loader.js:612:32)
03:04:07         at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
03:04:07         at Function.Module._load (internal/modules/cjs/loader.js:543:3)
03:04:07         at Function.Module.runMain (internal/modules/cjs/loader.js:744:10)
03:04:07         at startup (internal/bootstrap/node.js:261:19)
03:04:07         at bootstrapNodeJSCore (internal/bootstrap/node.js:595:3)
03:04:07   ...

Trott · 2018-06-04T08:24:06Z

I updated the title of this to remove on FreeBSD as it appears to be more widespread than that.

Trott · 2018-06-04T08:32:15Z

Stress test to confirm this is reproducible on FreeBSD:

https://ci.nodejs.org/job/node-stress-single-test/1906/nodes=freebsd11-x64/

Trott · 2018-06-04T09:36:24Z

stderr is empty and the assertion swallows all the other info that might be useful (status, stdout, signal).

PR to improve output: #21120

test-trace-events-fs-sync has been failing ocasinally on FreeBSD. While we don't have a fix, it should be marked as flaky. Ref: #21038 PR-URL: #21039 Refs: #21038 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Trivikram Kamat <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Anatoli Papirovski <[email protected]> Reviewed-By: Lance Ball <[email protected]>

maclover7 · 2018-06-06T21:11:12Z

Failure on linuxone:

19:19:16 not ok 1832 parallel/test-trace-events-fs-sync
19:19:16   ---
19:19:16   duration_ms: 1.458
19:19:16   severity: fail
19:19:16   exitcode: 1
19:19:16   stack: |-
19:19:16     assert.js:80
19:19:16       throw new AssertionError(obj);
19:19:16       ^
19:19:16     
19:19:16     AssertionError [ERR_ASSERTION]: fs.sync.fchmod: 
19:19:16         at Object.<anonymous> (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/parallel/test-trace-events-fs-sync.js:131:10)
19:19:16         at Module._compile (internal/modules/cjs/loader.js:702:30)
19:19:16         at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
19:19:16         at Module.load (internal/modules/cjs/loader.js:612:32)
19:19:16         at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
19:19:16         at Function.Module._load (internal/modules/cjs/loader.js:543:3)
19:19:16         at Function.Module.runMain (internal/modules/cjs/loader.js:744:10)
19:19:16         at startup (internal/bootstrap/node.js:261:19)
19:19:16         at bootstrapNodeJSCore (internal/bootstrap/node.js:595:3)
19:19:16   ...

apapirovski · 2018-06-06T21:44:48Z

Stress test now that we have more useful output: https://ci.nodejs.org/job/node-stress-single-test/1918/

apapirovski · 2018-06-07T13:49:40Z

So... we've got a SIGSEGV... :|

09:15:45 not ok 53 parallel/test-trace-events-fs-sync
09:15:45   ---
09:15:45   duration_ms: 0.340
09:15:45   severity: fail
09:15:45   exitcode: 1
09:15:45   stack: |-
09:15:45     assert.js:80
09:15:45       throw new AssertionError(obj);
09:15:45       ^
09:15:45     
09:15:45     AssertionError [ERR_ASSERTION]: fs.sync.access:
09:15:45     { status: null,
09:15:45       signal: 'SIGSEGV',
09:15:45       output: [ null, '', '' ],
09:15:45       pid: 83533,
09:15:45       stdout: '',
09:15:45       stderr: '' }
09:15:45         at Object.<anonymous> (/usr/home/iojs/build/workspace/node-stress-single-test/nodes/freebsd11-x64/test/parallel/test-trace-events-fs-sync.js:136:10)
09:15:45         at Module._compile (internal/modules/cjs/loader.js:702:30)
09:15:45         at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
09:15:45         at Module.load (internal/modules/cjs/loader.js:612:32)
09:15:45         at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
09:15:45         at Function.Module._load (internal/modules/cjs/loader.js:543:3)
09:15:45         at Function.Module.runMain (internal/modules/cjs/loader.js:744:10)
09:15:45         at startup (internal/bootstrap/node.js:267:19)
09:15:45         at bootstrapNodeJSCore (internal/bootstrap/node.js:600:3)

This could be related to the trace events failures we've previously seen on Windows. ping @jasnell @ofrobots since one of you previously mentioned some segfaults that you saw in the trace events code and were planning on fixing?

ofrobots · 2018-06-07T17:06:23Z

I'm investigating some SIGSEGVs in tracing today.

ofrobots · 2018-06-07T23:30:19Z

There are a few flaws in the choreography of the tracing event loop on shutdown that cause the segfault: basically the libuv threadpool mutex gets destroyed as part of the static destructors, but the tracing agent shutdown happens after that. It tries to do some IO to write the terminating JSON characters, which end up crashing. I'm still looking into what might be the best way to fix this.

This explains the segfault. It is not clear to me whether the other original failures are related to this same underlying issue, but first things first.

ofrobots · 2018-06-08T20:32:46Z

/cc @eugeneo

Trott · 2018-06-20T16:27:38Z

Failed on LinuxONE again today.

https://ci.nodejs.org/job/node-test-commit-linuxone/2386/nodes=rhel72-s390x/console

10:12:08 not ok 1848 parallel/test-trace-events-fs-sync
10:12:08   ---
10:12:08   duration_ms: 1.462
10:12:08   severity: fail
10:12:08   exitcode: 1
10:12:08   stack: |-
10:12:08     assert.js:80
10:12:08       throw new AssertionError(obj);
10:12:08       ^
10:12:08     
10:12:08     AssertionError [ERR_ASSERTION]: fs.sync.fstat:
10:12:08     { status: null,
10:12:08       signal: 'SIGSEGV',
10:12:08       output: [ null, '', '' ],
10:12:08       pid: 51890,
10:12:08       stdout: '',
10:12:08       stderr: '' }
10:12:08         at Object.<anonymous> (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/parallel/test-trace-events-fs-sync.js:136:10)
10:12:08         at Module._compile (internal/modules/cjs/loader.js:702:30)
10:12:08         at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
10:12:08         at Module.load (internal/modules/cjs/loader.js:612:32)
10:12:08         at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
10:12:08         at Function.Module._load (internal/modules/cjs/loader.js:543:3)
10:12:08         at Function.Module.runMain (internal/modules/cjs/loader.js:744:10)
10:12:08         at startup (internal/bootstrap/node.js:237:19)
10:12:08         at bootstrapNodeJSCore (internal/bootstrap/node.js:561:3)
10:12:08   ...

Trott · 2018-07-15T04:00:55Z

Failed again on LinuxONE today.

https://ci.nodejs.org/job/node-test-commit-linuxone/nodes=rhel72-s390x/3018/console

22:34:36 not ok 1910 parallel/test-trace-events-fs-sync
22:34:36   ---
22:34:36   duration_ms: 3.131
22:34:36   severity: fail
22:34:36   exitcode: 1
22:34:36   stack: |-
22:34:36     assert.js:80
22:34:36       throw new AssertionError(obj);
22:34:36       ^
22:34:36     
22:34:36     AssertionError [ERR_ASSERTION]: fs.sync.stat:
22:34:36     { status: null,
22:34:36       signal: 'SIGSEGV',
22:34:36       output: [ null, '', '' ],
22:34:36       pid: 60920,
22:34:36       stdout: '',
22:34:36       stderr: '' }
22:34:36         at Object.<anonymous> (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/parallel/test-trace-events-fs-sync.js:139:10)
22:34:36         at Module._compile (internal/modules/cjs/loader.js:689:30)
22:34:36         at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
22:34:36         at Module.load (internal/modules/cjs/loader.js:599:32)
22:34:36         at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
22:34:36         at Function.Module._load (internal/modules/cjs/loader.js:530:3)
22:34:36         at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
22:34:36         at startup (internal/bootstrap/node.js:266:19)
22:34:36         at bootstrapNodeJSCore (internal/bootstrap/node.js:596:3)
22:34:36   ...

Trott · 2018-08-05T05:01:07Z

LinuxONE again:

https://ci.nodejs.org/job/node-test-commit-linuxone/nodes=rhel72-s390x/3551/console

21:48:50 not ok 1954 parallel/test-trace-events-fs-sync
21:48:50   ---
21:48:50   duration_ms: 2.975
21:48:50   severity: fail
21:48:50   exitcode: 1
21:48:50   stack: |-
21:48:50     assert.js:84
21:48:50       throw new AssertionError(obj);
21:48:50       ^
21:48:50     
21:48:50     AssertionError [ERR_ASSERTION]: fs.sync.open:
21:48:50     { status: null,
21:48:50       signal: 'SIGSEGV',
21:48:50       output: [ null, '', '' ],
21:48:50       pid: 24620,
21:48:50       stdout: '',
21:48:50       stderr: '' }
21:48:50         at Object.<anonymous> (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/parallel/test-trace-events-fs-sync.js:139:10)
21:48:50         at Module._compile (internal/modules/cjs/loader.js:689:30)
21:48:50         at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
21:48:50         at Module.load (internal/modules/cjs/loader.js:599:32)
21:48:50         at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
21:48:50         at Function.Module._load (internal/modules/cjs/loader.js:530:3)
21:48:50         at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
21:48:50         at startup (internal/bootstrap/node.js:266:19)
21:48:50         at bootstrapNodeJSCore (internal/bootstrap/node.js:596:3)
21:48:50   ...

Original commit message: [tracing] do not add traces when disabled #21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <[email protected]> Reviewed-by: Yang Guo <[email protected]> Cr-Commit-Position: refs/heads/master@{#55809} Refs: v8/v8@2363cdf PR-URL: #22812 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Eugene Ostroukhov <[email protected]>

ofrobots · 2018-09-14T01:15:27Z

#22812 has landed. Hopefully it fixes it. There might still be some other thread races, but I haven't been able to observe any so far in my testing.

Trott · 2018-09-14T05:11:17Z

@ofrobots 🎉 Thank you so much!

I'll open a PR to unmark the test as flaky and close this issue.

Original commit message: [tracing] do not add traces when disabled #21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <[email protected]> Reviewed-by: Yang Guo <[email protected]> Cr-Commit-Position: refs/heads/master@{#55809} Refs: v8/v8@2363cdf PR-URL: #22812 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Eugene Ostroukhov <[email protected]>

Trott · 2018-09-14T17:19:46Z

It's flaky still, but in a different way (SIGABRT rather than SIGSEGV). Will open a new issue, but ref this one...

Original commit message: [tracing] do not add traces when disabled nodejs#21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <[email protected]> Reviewed-by: Yang Guo <[email protected]> Cr-Commit-Position: refs/heads/master@{nodejs#55809} Refs: v8/v8@2363cdf

Original commit message: [tracing] do not add traces when disabled #21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <[email protected]> Reviewed-by: Yang Guo <[email protected]> Cr-Commit-Position: refs/heads/master@{#55809} Refs: v8/v8@2363cdf PR-URL: #22812 Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Eugene Ostroukhov <[email protected]>

Original commit message: [tracing] do not add traces when disabled nodejs#21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <[email protected]> Reviewed-by: Yang Guo <[email protected]> Cr-Commit-Position: refs/heads/master@{nodejs#55809} Refs: v8/v8@2363cdf

Original commit message: [tracing] do not add traces when disabled #21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <[email protected]> Reviewed-by: Yang Guo <[email protected]> Cr-Commit-Position: refs/heads/master@{#55809} Refs: v8/v8@2363cdf PR-URL: #22754 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Refael Ackermann <[email protected]> Reviewed-By: Ali Ijaz Sheikh <[email protected]> Reviewed-By: Colin Ihrig <[email protected]>

nodejs/node#21038 Change-Id: Ic4c9f403b5e54a97d3170b2311dd5aab8c8357c8 Reviewed-on: https://chromium-review.googlesource.com/1217726 Commit-Queue: Ali Ijaz Sheikh <[email protected]> Reviewed-by: Yang Guo <[email protected]> Cr-Commit-Position: refs/heads/master@{#55809}

Unreliability for test-trace-events-fs-sync is believed to have been fixed. Remove flaky designation. Ref: nodejs#22812 Ref: nodejs#21038 (comment)

Unreliability for test-trace-events-fs-sync is believed to have been fixed. Remove flaky designation. Ref: nodejs#22812 Ref: nodejs#21038 (comment) PR-URL: nodejs#22856 Refs: nodejs#22812 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Weijia Wang <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Sakthipriyan Vairamani <[email protected]> Reviewed-By: Yuta Hiroto <[email protected]> Reviewed-By: Refael Ackermann <[email protected]>

Unreliability for test-trace-events-fs-sync is believed to have been fixed. Remove flaky designation. Ref: #22812 Ref: #21038 (comment) PR-URL: #22856 Refs: #22812 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Weijia Wang <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Sakthipriyan Vairamani <[email protected]> Reviewed-By: Yuta Hiroto <[email protected]> Reviewed-By: Refael Ackermann <[email protected]>

mmarchini added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label May 30, 2018

mmarchini changed the title ~~Flaky test-trace-events-fs-sync on FreeBSD~~ Flaky parallel/test-trace-events-fs-sync on FreeBSD May 30, 2018

mmarchini pushed a commit to mmarchini/node that referenced this issue May 30, 2018

test: mark test-trace-events-fs-sync as flaky

f6aa479

test-trace-events-fs-sync has been failing ocasinally on FreeBSD. While we don't have a fix, it should be marked as flaky. Ref: nodejs#21038

mmarchini mentioned this issue May 30, 2018

test: mark test-trace-events-fs-sync as flaky #21039

Closed

1 task

mmarchini mentioned this issue May 31, 2018

deps: update V8 to 6.7 #19989

Closed

2 tasks

Trott changed the title ~~Flaky parallel/test-trace-events-fs-sync on FreeBSD~~ Flaky parallel/test-trace-events-fs-sync Jun 4, 2018

ofrobots self-assigned this Jun 7, 2018

addaleax mentioned this issue Jun 25, 2018

src: remove extra ReadStop() call #21528

Closed

2 tasks

addaleax added the trace_events Issues and PRs related to V8, Node.js core, and userspace code trace events. label Jul 16, 2018

addaleax self-assigned this Jul 17, 2018

ofrobots mentioned this issue Sep 12, 2018

tracing: Fix some race conditions #22812

Closed

2 tasks

Trott closed this as completed Sep 14, 2018

Trott mentioned this issue Sep 14, 2018

test: remove flaky designation for test #22856

Closed

2 tasks

Trott mentioned this issue Sep 14, 2018

Investigate flaky test/parallel/test-trace-events-fs-sync #22865

Closed

Trott added a commit to Trott/io.js that referenced this issue Oct 6, 2018

test: remove flaky designation for test

0b924f4

Unreliability for test-trace-events-fs-sync is believed to have been fixed. Remove flaky designation. Ref: nodejs#22812 Ref: nodejs#21038 (comment)

refack mentioned this issue Oct 18, 2018

build: spawn make test-ci with -j1 #23733

Merged

3 tasks

F3n67u mentioned this issue Jun 20, 2022

Investigate flaky test-trace-events-fs-sync #43502

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flaky parallel/test-trace-events-fs-sync #21038

Flaky parallel/test-trace-events-fs-sync #21038

mmarchini commented May 30, 2018

Trott commented May 31, 2018

Trott commented Jun 4, 2018 •

edited

Loading

Trott commented Jun 4, 2018

Trott commented Jun 4, 2018 •

edited

Loading

Trott commented Jun 4, 2018

maclover7 commented Jun 6, 2018

apapirovski commented Jun 6, 2018 •

edited

Loading

apapirovski commented Jun 7, 2018

ofrobots commented Jun 7, 2018

ofrobots commented Jun 7, 2018 •

edited

Loading

ofrobots commented Jun 8, 2018

Trott commented Jun 20, 2018

Trott commented Jul 15, 2018

Trott commented Aug 5, 2018

ofrobots commented Sep 14, 2018

Trott commented Sep 14, 2018

Trott commented Sep 14, 2018

Flaky parallel/test-trace-events-fs-sync #21038

Flaky parallel/test-trace-events-fs-sync #21038

Comments

mmarchini commented May 30, 2018

Trott commented May 31, 2018

Trott commented Jun 4, 2018 • edited Loading

Trott commented Jun 4, 2018

Trott commented Jun 4, 2018 • edited Loading

Trott commented Jun 4, 2018

maclover7 commented Jun 6, 2018

apapirovski commented Jun 6, 2018 • edited Loading

apapirovski commented Jun 7, 2018

ofrobots commented Jun 7, 2018

ofrobots commented Jun 7, 2018 • edited Loading

ofrobots commented Jun 8, 2018

Trott commented Jun 20, 2018

Trott commented Jul 15, 2018

Trott commented Aug 5, 2018

ofrobots commented Sep 14, 2018

Trott commented Sep 14, 2018

Trott commented Sep 14, 2018

Trott commented Jun 4, 2018 •

edited

Loading

Trott commented Jun 4, 2018 •

edited

Loading

apapirovski commented Jun 6, 2018 •

edited

Loading

ofrobots commented Jun 7, 2018 •

edited

Loading