Operator RandomObjectBBox #2657

mzient · 2021-02-04T23:06:13Z

Why we need this PR?

Pick one, remove the rest

It adds new feature needed for UNet: select bounding box of a random blob from randomly selected foreground class

What happened in this PR?

Fill relevant points, put NA otherwise. Replace anything inside []

What solution was applied:
- Check what classes are present
- Select random class according to weights (if given)
- Filter the input
- Find blobs
- Find boxes
- Apply threshold and k-largest - if no good blobs found, try other classes
- If no class is good, return entire input area
Affected modules and functionalities:
- LabelBBox - bugfixes
Key points relevant for the review:
- Argument handling
Validation and testing:
- Python tests
Documentation (including examples):
- Docstrings

JIRA TASK: DALI-1817

mzient · 2021-02-08T23:24:02Z

!build

dali-automaton · 2021-02-08T23:25:59Z

CI MESSAGE: [2056699]: BUILD STARTED

lgtm-com · 2021-02-08T23:33:41Z

This pull request introduces 1 alert when merging 5402c7dad5d57cca5a0cf3e47658c4aae626682f into 2c0b597 - view on LGTM.com

new alerts:

1 for Unreachable code

dali-automaton · 2021-02-09T03:40:24Z

CI MESSAGE: [2056699]: BUILD FAILED

mzient · 2021-02-09T09:00:28Z

!build

dali-automaton · 2021-02-09T09:05:52Z

CI MESSAGE: [2058334]: BUILD STARTED

dali-automaton · 2021-02-09T10:22:15Z

CI MESSAGE: [2058334]: BUILD FAILED

mzient · 2021-02-09T13:35:50Z

!build

dali-automaton · 2021-02-09T13:39:41Z

CI MESSAGE: [2058874]: BUILD STARTED

dali-automaton · 2021-02-09T13:40:40Z

CI MESSAGE: [2058877]: BUILD STARTED

dali-automaton · 2021-02-09T15:16:15Z

CI MESSAGE: [2058877]: BUILD PASSED

dali-automaton · 2021-02-09T16:56:36Z

CI MESSAGE: [2058874]: BUILD PASSED

mzient · 2021-02-09T17:04:31Z

dali/operators/segmentation/random_object_bbox.cc

+        context.classes.push_back(cls);
+        context.weights.push_back(1);
+      }
+      std::sort(context.classes.begin(), context.classes.end());


Sorting is necessary, because the order of elements in unordered set is not only undefined with respect to current contents, but also depends on previous contents (before clear) - as this changes the number of bins.

It's cheaper to copy it from unordered set and sort, because unordered set has better memory management than set and we need a copy to a flat array anyway.

I would be for writing such comments directly in the code instead of the PR review.

jantonguirao · 2021-02-10T09:35:49Z

dali/operators/segmentation/random_object_bbox.cc

+This operator takes a labeled segmentation map as its input. With probability ``foreground_prob``
+it randomly selects a label (uniformly or according to the distribution given as ``class_weights``),
+extracts connected blobs of pixels with the selected label and randomly selects one of them
+(with additional constraints given as ``k_largest`` and ``threshold``).


I feel the content of this parenthesis deserves its own sentence. It is not clear enough now.

I'll try to rephrase it.

jantonguirao · 2021-02-10T09:35:58Z

dali/operators/segmentation/random_object_bbox.cc

+it randomly selects a label (uniformly or according to the distribution given as ``class_weights``),
+extracts connected blobs of pixels with the selected label and randomly selects one of them
+(with additional constraints given as ``k_largest`` and ``threshold``).
+With probability 1-foreground_prob, entire area of the input is returned.)")


Suggested change

With probability 1-foreground_prob, entire area of the input is returned.)")

With probability 1-foreground_prob, the entire area of the input is returned.)")

Maybe also add info of what are the outputs (and possible formats?)

I don't know if it's wort repeating here.... I'll take a look.

dali/operators/segmentation/random_object_bbox.cc

jantonguirao · 2021-02-10T09:38:50Z

dali/operators/segmentation/random_object_bbox.cc

+    return 1 + separate_corners + output_class;
+  })
+  .AddOptionalArg("ignore_class", R"(If True, all objects are picked with equal probability,
+regardless of the class they belong to. Otherwise, a class is picked first and then object is


Suggested change

regardless of the class they belong to. Otherwise, a class is picked first and then object is

regardless of the class they belong to. Otherwise, a class is picked first and then the object is

an object

jantonguirao · 2021-02-10T09:40:06Z

dali/operators/segmentation/random_object_bbox.cc

+  This flag only affects the probability with which blobs are selected. It does not cause
+  blobs of different classes to be merged.)", false)
+  .AddOptionalArg("output_class", R"(If True, an additional output is produced which contains the
+label of the class to which the resulting box belongs, or background label if foreground box


Suggested change

label of the class to which the resulting box belongs, or background label if foreground box

label of the class to which the resulting box belongs, or the background label if the selected box

dali/operators/segmentation/random_object_bbox.cc

jantonguirao · 2021-02-10T10:18:22Z

dali/operators/segmentation/random_object_bbox.cc

+void RandomObjectBBox::GetClassesAndWeightsArgs(
+      ClassVec &classes, WeightVec &weights, int &background, int sample_idx) {
+  background = background_[sample_idx].data[0];
+  if (ignore_class_)


I would rather put this if before calling this function.

But then I'd have to get background in the calling function.

Yes, or have a separate getter for the background so that you see:

GetBackground(...) if (!ignore_label_) GetClassesAndWeights(...)

It doesn't work like this. If there are classes or weights, background may depend on them.

jantonguirao · 2021-02-10T10:18:50Z

dali/operators/segmentation/random_object_bbox.cc

+void StoreBox(const OutListCPU<int, 1> &out1,
+              const OutListCPU<int, 1> &out2,
+              RandomObjectBBox::OutputFormat format,
+              int sample_idx, Box &&box) {


Just a question: Have you considered using one of the existing structures for Box/ROI/etc?

These structures have fixed number of dimensions whereas this works with dynamic dimensions.

jantonguirao · 2021-02-10T10:21:56Z

dali/operators/segmentation/random_object_bbox.cc

+    }
+    if (choose_different_background) {
+      // This will yield incorrect result if the set of classes contains both INT_MIN and INT_MAX,
+      // which is a sacrifice I am willing to make.


can you enforce somewhere that this is not the case?

I could even loop over and decrement until I find a free slot.... I don't know how much value is there in it.

I was thinking of a simple assert or DALI_ENFORCE.

dali/operators/segmentation/random_object_bbox.cc

mzient · 2021-02-10T16:30:28Z

dali/operators/segmentation/random_object_bbox.cc

+  .AddOptionalArg<vector<int>>("threshold", R"(Minimum extent(s) of the bounding boxes to return.
+
+If current class doesn't contain any bounding box that meets this condition, the largest one
+is returned.)", nullptr, true)


Hmmm... I see that I didn't implement that (and just try a different class or fall back to background).
I don't know what I should do in this case. Change implementation or change the docs? Arguably, it makes sense not to return a box below the threshold, even if it's the only one in its class.

I am OK with that (not returning the box). Given that it is properly documented

Now the documentation follows what happens in the code.

jantonguirao · 2021-02-11T08:18:48Z

dali/operators/segmentation/random_object_bbox.cc

+    SmallVector<std::pair<int64_t, int>, 32> vol_idx;
+    vol_idx.resize(n);
+    for (int i = 0; i < n; i++) {
+      vol_idx[i] = { -volume(boxes[i]), i };


You could leave the volume possitive and use std::greater<> with std::sort

But then I'd have to reverse sign of i to get stability.

jantonguirao · 2021-02-11T08:20:29Z

dali/operators/segmentation/random_object_bbox.cc

+  if (n <= 0)
+    return -1;
+
+  if (k_largest_ > 0) {


Small improvement: If your k_largest is bigger than the actual number of objects you don't need to sort either.

jantonguirao · 2021-02-11T08:21:23Z

dali/operators/segmentation/random_object_bbox.cc

+
+  VALUE_SWITCH(ndim, static_ndim, (1, 2, 3, 4, 5, 6),
+    (
+      auto *box_data = reinterpret_cast<Box<static_ndim, int>*>(ctx.box_data.data());


Wasn't it technically undefined behavior? :)

I know, but it's sort-of less undefined here, since it crosses function boundary and it's a tad safer to type-pun an array instead of one object.

jantonguirao · 2021-02-11T08:27:15Z

dali/operators/segmentation/random_object_bbox.cc

+template <typename T>
+bool RandomObjectBBox::PickForegroundBox(
+      SampleContext &context, const InTensorCPU<T> &input) {
+  GetClassesAndWeightsArgs(context.classes, context.weights, context.background,


I'd move this call inside if (ignore_class_) and remove the check inside GetClassesAndWeights? Maybe have a separate getter for the background

jantonguirao · 2021-02-11T08:29:02Z

dali/operators/segmentation/random_object_bbox.cc

+      }
+      std::sort(context.classes.begin(), context.classes.end());
+    } else {
+      for (int i = 0; i < static_cast<int>(context.classes.size()); i++) {


Suggested change

for (int i = 0; i < static_cast<int>(context.classes.size()); i++) {

for (size_t i = 0; i < context.classes.size(); i++) {

jantonguirao · 2021-02-11T08:45:40Z

dali/operators/segmentation/random_object_bbox.cc

+    class_label_out = view<int, 0>(ws.OutputRef<CPUBackend>(class_output_idx_));
+
+  TensorShape<> default_anchor;
+  default_anchor.resize(ndim);


Suggested change

default_anchor.resize(ndim);

default_anchor.resize(ndim, 0);

jantonguirao · 2021-02-11T08:46:20Z

dali/operators/segmentation/random_object_bbox.cc

+
+        if (HasClassLabelOutput())
+          class_label_out.data[i][0] = ctx.class_label;
+      }, volume(in_shape.tensor_shape_span(i)));


Suggested change

}, volume(in_shape.tensor_shape_span(i)));

}, in_shape.tensor_size(i));

jantonguirao · 2021-02-11T08:48:17Z

dali/operators/segmentation/random_object_bbox.h

+
+    bool output_class_ = spec.GetArgument<bool>("output_class");
+
+    if (ignore_class_ && (classes_.IsDefined() || weights_.IsDefined() || output_class_)) {


OK, so if you have this, you can simply place an assert(!ignore_class) in the function where you get classes and weights

Then I'd have to duplicate the code to get background value - and the control flow doesn't favor it.

jantonguirao · 2021-02-11T10:15:35Z

dali/test/python/test_operator_random_object_bbox.py

+
+                format = formats[fmt]
+                fmt = (fmt + 1) % len(formats)
+                dtype = types[np.random.randint(0, len(types))]


Use np.random.choice(types)

Actually, now numpy complain that random choice is deprecated for non-numeric types... I'd leave it as-is and fix it in the parallel PR...?

jantonguirao · 2021-02-11T10:15:59Z

dali/test/python/test_operator_random_object_bbox.py

+
+    format = format or "anchor_shape"
+
+    for _ in range(50):


can you make the number of iterations a parameter of the function?

I could, but what for?

It's easy to change later if we need to

mzient · 2021-02-11T15:39:58Z

!build

dali-automaton · 2021-02-11T15:46:13Z

CI MESSAGE: [2067103]: BUILD STARTED

dali-automaton · 2021-02-11T17:41:02Z

CI MESSAGE: [2067103]: BUILD PASSED

mzient · 2021-02-12T09:29:40Z

!build

dali-automaton · 2021-02-12T09:36:08Z

CI MESSAGE: [2070401]: BUILD STARTED

dali-automaton · 2021-02-12T11:08:01Z

CI MESSAGE: [2070401]: BUILD PASSED

Signed-off-by: Michał Zientkiewicz <[email protected]>

* Propagate batch size from batch external sources to per-sample external sources * Fix bugs Signed-off-by: Michał Zientkiewicz <[email protected]>

* Add `first = False` in external source handling to produce an error when external sources disagree on batch size. Signed-off-by: Michał Zientkiewicz <[email protected]>

Signed-off-by: Michał Zientkiewicz <[email protected]>

mzient · 2021-02-15T18:33:20Z

!build

dali-automaton · 2021-02-15T18:35:38Z

CI MESSAGE: [2077424]: BUILD STARTED

dali-automaton · 2021-02-15T20:07:04Z

CI MESSAGE: [2077424]: BUILD PASSED

Signed-off-by: Michał Zientkiewicz <[email protected]>

mzient · 2021-02-16T10:46:27Z

!build

dali-automaton · 2021-02-16T10:51:56Z

CI MESSAGE: [2078838]: BUILD STARTED

dali-automaton · 2021-02-16T12:36:53Z

CI MESSAGE: [2078838]: BUILD PASSED

klecki

I'm yet to review the tests, but I'm already done with most of the implementation so posting my comments.
The flow looks correct but I had objections to the SampleContext and its API: mainly the fact that the Init will leave it in some weird state and the GetBgFgAndWeights is done a bit from the outside (even though it probably shouldn't access the ArgValues directly, it would make more sense to me if the initialization was done first and left the object in a complete state).

klecki · 2021-02-16T15:37:56Z

dali/operators/segmentation/random_object_bbox.h

+    format_ = ParseOutputFormat(spec.GetArgument<string>("format"));
+    ignore_class_ = spec.GetArgument<bool>("ignore_class");
+
+    bool output_class_ = spec.GetArgument<bool>("output_class");


May be misleading someone into thinking that it is a member variable.

Suggested change

bool output_class_ = spec.GetArgument<bool>("output_class");

bool output_class = spec.GetArgument<bool>("output_class");

klecki · 2021-02-16T16:52:09Z

dali/operators/segmentation/random_object_bbox.cc

+      StoreBox(out1, out2, format_, i, default_anchor, input.tensor_shape(i));
+      if (HasClassLabelOutput()) {
+        SampleContext &ctx = contexts_[0];
+        GetBgFgAndWeights(ctx.classes, ctx.weights, ctx.background, i);


Don't you need to just output the background for this sample. Is the whole GetBgFgAndWeights necessary here? There won't be any further processing done for this sample, right?

Background label may be automatically assigned so as to skip classes, so I need to process them anyway.

klecki · 2021-02-16T18:04:19Z

dali/operators/segmentation/random_object_bbox.cc

+  StoreBox(out1, out2, format, sample_idx, box.lo, box.hi);
+}
+
+void RandomObjectBBox::GetBgFgAndWeights(


As I'm reading this I feel a bit of spaghetti. The context has Init, but it only resizes some data, than there is this function, which directly modifies context's members (but if ignore_class_ is true I guess it will just leave them in an undefined state, either some leftovers from previous sample or just empty), than the context again has methods to operate on for example weights, assuming that they come from outside and are valid (and it uses them to further initialize some members).

I'm thinking how to untangle this, maybe I would start with zeroing the classes and weights in the context Init or having the Init inspect the classes and weights for that input and initialize those (so do more of GetBgFgAndWeights).

I admit that it is indeed spaghettish, but I'd prefer to address this in the followup (parallelization PR), as changing this logic here would result in very painful rebasing there - the context object has undergone significant changes between the two.

I can understand that, if you can take of untangling it a bit in the follow up I would be ok with that under current circumstances.

klecki · 2021-02-16T18:29:30Z

dali/operators/segmentation/random_object_bbox.cc

+        }
+
+        if (HasClassLabelOutput())
+          class_label_out.data[i][0] = ctx.class_label;


This result passed via ctx from the two possible branches of the if (PickForegroundBox(ctx)) feels a bit weird.
Maybe encapsulate it in StoreLabel(i, ctx.selected_label)/StoreLabel(i, ctx.background) in the if/else respectively.

I don't think makes much sense to encapsulate a simple assignment in a function - especially given that I'd either have to pass (ugly) or store in the object (uglier) the class_label_out variable, which is now local.

klecki · 2021-02-16T18:55:45Z

dali/operators/segmentation/random_object_bbox.cc

+
+    context.labels.erase(context.background);
+
+    if (!classes_.IsDefined() && !weights_.IsDefined()) {


This is the last case, for the GetBgFgAndWeights, but it's not handled there. I understand that you need to gather the labels first, but than, why not gather them in the Init? You're already inspecting the input a bit.

Init is called in the main thread and I definitely don't want to scan entire input there.

For the fg case (else), where you add the tasks to thread pool it's already in the lambda and I don't see any other invocation.

Indeed. However, I don't need to call init at all for the background case - I don't need the temporary storage and allocating 100s of megabytes of arrays that would be never used is not something I'd do lightly.

klecki · 2021-02-16T18:56:55Z

dali/operators/segmentation/random_object_bbox.cc

+      context.classes.clear();
+      context.weights.clear();
+      for (auto cls : context.labels) {
+        context.classes.push_back(cls);


The above comment is because this looked really familiar to 230-231.

klecki · 2021-02-17T11:44:42Z

dali/operators/segmentation/random_object_bbox.cc

+      GetLabelBoundingBoxes(boxes, ctx.blobs.to_static<static_ndim>(), -1);
+      int box_idx = PickBox(boxes, ctx.sample_idx);
+      if (box_idx >= 0) {
+        ctx.SelectBox(box_idx);


If you already did the reinterpret why not:

Suggested change

ctx.SelectBox(box_idx);

ctx.SelectBox(boxes[box_idx]);

klecki · 2021-02-17T12:05:37Z

dali/operators/segmentation/random_object_bbox.cc

+    assert(threshold_.get().shape[sample_idx] == TensorShape<1>{ ndim });
+    for (int i = 0; i < ndim; i++)
+      threshold[i] = thresh[i];
+    end = std::remove_if(beg, end, [threshold](auto box) {


Any difference if the auto box was a reference instead of pass by value?

It probably should be.

klecki

I focused mostly on the test coverage (and didn't dig that deep into generating the test data and results in python), so mostly suggestions for some more testing that could be done.

klecki · 2021-02-17T12:40:19Z

dali/test/python/test_operator_random_object_bbox.py

+
+def test_num_output():
+    """Test that a proper number of outputs is produced, depending on arguments"""
+    inp = fn.external_source(data, batch=False)


I was looking where the data comes from, and it surprised me below. Could you maybe move it to the top?

klecki · 2021-02-17T14:51:02Z

dali/test/python/test_operator_random_object_bbox.py

+                threshold_opt = [None, 3, list(range(1, 1+ndim)), fn.random.uniform(range=(1,5), shape=[ndim], dtype=dali.types.INT32, seed=13231)]
+                threshold = np.random.choice(threshold_opt);
+                k_largest_opt = [None, 1, 2, 5]
+                k_largest = np.random.choice(k_largest_opt);


I'm not sure if having threshold and k_largest as randomly tested is a good decision, probably it would be safer to make sure that they are tested with both unspecified, one of them specified and both specified.

To not have have cartesian product of all possibilities, you could probably test that for a praticular setting of classes etc.

klecki · 2021-02-17T14:57:26Z

dali/test/python/test_operator_random_object_bbox.py

+
+    return label
+
+def batch_generator(batch_size, ndim, dtype):


Would it make sense to test with data that would give deterministic boxes as a result, for example have classes with 1 box, and set the weight for one of them to 1 and rest to 0?

That's requesting 100s extra lines of tests for something that's pretty well covered by existing tests.

klecki · 2021-02-17T15:01:40Z

dali/test/python/test_operator_random_object_bbox.py

+from test_utils import check_batch
+from test_utils import np_type_to_dali
+from nose.tools import raises, assert_raises, nottest
+


I would maybe consider adding negative tests, stuff like not matching number of weights and classes passed should raise, etc. and if the ignore_classes is specified other class-related arguments should cause an error.

Signed-off-by: Michał Zientkiewicz <[email protected]>

dali-automaton · 2021-02-17T19:47:15Z

CI MESSAGE: [2083639]: BUILD STARTED

dali-automaton · 2021-02-17T21:46:20Z

CI MESSAGE: [2083639]: BUILD PASSED

mzient requested a review from a team February 4, 2021 23:06

jantonguirao assigned jantonguirao and awolant Feb 8, 2021

mzient changed the title ~~[WIP] Operator RandomObjectBBox~~ Operator RandomObjectBBox Feb 8, 2021

mzient force-pushed the RandomObjectBBoxOp branch from a3a912a to d99fa00 Compare February 9, 2021 08:59

jantonguirao assigned klecki Feb 9, 2021

mzient commented Feb 9, 2021

View reviewed changes

jantonguirao reviewed Feb 10, 2021

View reviewed changes

mzient commented Feb 10, 2021

View reviewed changes

jantonguirao reviewed Feb 11, 2021

View reviewed changes

mzient force-pushed the RandomObjectBBoxOp branch 2 times, most recently from 49fe8d8 to 2ecd968 Compare February 12, 2021 09:28

mzient added 12 commits February 15, 2021 19:19

Initial effort for RandomObjectBBox

f688b0c

Signed-off-by: Michał Zientkiewicz <[email protected]>

[WIP]

3334f14

Signed-off-by: Michał Zientkiewicz <[email protected]>

It compiles!

468036a

Signed-off-by: Michał Zientkiewicz <[email protected]>

More or less working.

60e55f4

Signed-off-by: Michał Zientkiewicz <[email protected]>

Add extra argument with with class label.

5e79cfe

Signed-off-by: Michał Zientkiewicz <[email protected]>

* Add tests

38b8894

* Propagate batch size from batch external sources to per-sample external sources * Fix bugs Signed-off-by: Michał Zientkiewicz <[email protected]>

* Fix numpy seeds in tests for reproducibility

75e608b

* Add `first = False` in external source handling to produce an error when external sources disagree on batch size. Signed-off-by: Michał Zientkiewicz <[email protected]>

Sort auto-generated labels.

f9a680e

Signed-off-by: Michał Zientkiewicz <[email protected]>

Address review issues.

ec1e391

Signed-off-by: Michał Zientkiewicz <[email protected]>

Review issues. Add a test for background label selection.

922596a

Signed-off-by: Michał Zientkiewicz <[email protected]>

Adjust to improved TensorVector API.

81a5451

Signed-off-by: Michał Zientkiewicz <[email protected]>

Review issues.

e734678

Signed-off-by: Michał Zientkiewicz <[email protected]>

mzient force-pushed the RandomObjectBBoxOp branch from 2ecd968 to e734678 Compare February 15, 2021 18:33

Remove changes to ExternalSource from this PR.

33fc46a

Signed-off-by: Michał Zientkiewicz <[email protected]>

klecki reviewed Feb 17, 2021

View reviewed changes

jantonguirao approved these changes Feb 17, 2021

View reviewed changes

klecki reviewed Feb 17, 2021

View reviewed changes

mzient added 2 commits February 17, 2021 18:27

Some review issues.

b2ae570

Signed-off-by: Michał Zientkiewicz <[email protected]>

Add tests for misconfiguration errors.

1adc635

Signed-off-by: Michał Zientkiewicz <[email protected]>

klecki approved these changes Feb 17, 2021

View reviewed changes

mzient merged commit 665d9ab into NVIDIA:master Feb 18, 2021

JanuszL mentioned this pull request May 19, 2021

DALI 2021 roadmap #2978

Closed

	With probability 1-foreground_prob, entire area of the input is returned.)")
	With probability 1-foreground_prob, the entire area of the input is returned.)")

	regardless of the class they belong to. Otherwise, a class is picked first and then object is
	regardless of the class they belong to. Otherwise, a class is picked first and then the object is

	label of the class to which the resulting box belongs, or background label if foreground box
	label of the class to which the resulting box belongs, or the background label if the selected box

	for (int i = 0; i < static_cast<int>(context.classes.size()); i++) {
	for (size_t i = 0; i < context.classes.size(); i++) {

	}, volume(in_shape.tensor_shape_span(i)));
	}, in_shape.tensor_size(i));


		bool output_class_ = spec.GetArgument<bool>("output_class");

		if (ignore_class_ && (classes_.IsDefined() \|\| weights_.IsDefined() \|\| output_class_)) {

	bool output_class_ = spec.GetArgument<bool>("output_class");
	bool output_class = spec.GetArgument<bool>("output_class");


		context.labels.erase(context.background);

		if (!classes_.IsDefined() && !weights_.IsDefined()) {

Operator RandomObjectBBox #2657

Operator RandomObjectBBox #2657

Conversation

mzient commented Feb 4, 2021 • edited Loading

Why we need this PR?

What happened in this PR?

mzient commented Feb 8, 2021

dali-automaton commented Feb 8, 2021

lgtm-com bot commented Feb 8, 2021

dali-automaton commented Feb 9, 2021

mzient commented Feb 9, 2021

dali-automaton commented Feb 9, 2021

dali-automaton commented Feb 9, 2021

mzient commented Feb 9, 2021

dali-automaton commented Feb 9, 2021

dali-automaton commented Feb 9, 2021

dali-automaton commented Feb 9, 2021

dali-automaton commented Feb 9, 2021

mzient Feb 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jantonguirao Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzient Feb 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzient Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzient commented Feb 11, 2021

dali-automaton commented Feb 11, 2021

dali-automaton commented Feb 11, 2021

mzient commented Feb 12, 2021

dali-automaton commented Feb 12, 2021

dali-automaton commented Feb 12, 2021

mzient commented Feb 15, 2021

dali-automaton commented Feb 15, 2021

dali-automaton commented Feb 15, 2021

mzient commented Feb 16, 2021

dali-automaton commented Feb 16, 2021

dali-automaton commented Feb 16, 2021

klecki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzient commented Feb 4, 2021 •

edited

Loading

mzient Feb 9, 2021 •

edited

Loading

jantonguirao Feb 11, 2021 •

edited

Loading

mzient Feb 15, 2021 •

edited

Loading

mzient Feb 11, 2021 •

edited

Loading

mzient Feb 17, 2021 •

edited

Loading

mzient Feb 17, 2021 •

edited

Loading