[RLlib] By-pass Evaluation workers when doing OPE #30135

kouroshHakha · 2022-11-09T08:32:29Z

Why are these changes needed?

This PR covers two things:

It fixes a bug in our Doubly Robust and Weighted Importance Sampling OPE methods. The normalization factor should be the mean of the weights across all the rows within a dataset not a given batch of that dataset. So we have to compute this statistic on the entire dataset first so that we can normalize the individual batch of scores.
When we do offline Evaluation it is better to use datasets directly instead of passing their shards to individual rollout workers so that they can sample. Using ray dataset api and bypassing evaluation workersets, in case of OPE, brings a couple of advantages:

Evaluation config would look much simpler and more transparent. The user does not have to set the evaluation_duration_unit = "episode" , and evaluation_duration=int(validation_len * num_workers // batch_size) anymore. As long as you give us a dataset, the evaluation will be done on that entire dataset after each training iteration is done (even if the dataset size is not integer multiple of the batch size // number of workers)
It is much faster and scales linearly with more resources. The previous solution for some reason did not scale well with more resources thrown at it. This one does. It scales so much that on a 96 core machine I can run WIS on a dataset of 2.6 M rows in < 20 seconds. This is insane compared to what we had before which was 600-2000 seconds (and also not consistent)

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

…e_on_dataset Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha · 2022-11-09T08:35:11Z

@sven1977 @gjoliver I'll clean up the PR tomorrow. In the meantime please go over the high-levels of the proposed changes. I'll ping you when the PR is ready for a more detailed review.

gjoliver

at a high level, I would expect to see some updates to the resource planning util here
https://github.com/gjoliver/ray/blob/master/rllib/offline/dataset_reader.py#L23
we reserve a bit of CPU resources for train dataset here.
maybe we should reserve some for the eval dataset as well, if we want to do things this way.
other than that, this seems like a much better approach than our existing single threaded eval logic.
nice!

gjoliver · 2022-11-09T09:32:15Z

rllib/algorithms/algorithm.py

+            # worker anymore
+            logger.info("Creating evaluation dataset ...")
+            self.evaluation_dataset, _ = get_dataset_and_shards(
+                self.evaluation_config, num_workers=0


comment that num_workers=0 is quite important. this will return un-sharded dataset as-is.

already did

sven1977 · 2022-11-09T10:53:47Z

rllib/algorithms/algorithm.py

@@ -616,7 +616,8 @@ def setup(self, config: AlgorithmConfig) -> None:

        # Evaluation WorkerSet setup.
        # User would like to setup a separate evaluation worker set.
-        if self.config.evaluation_num_workers > 0 or self.config.evaluation_interval:
+        # Note: We skipp workerset creation if we need to do offline evaluation


Dumb question: Is there any chance someone might want to do both? offline estimation on a dataset (one run through the complete dataset per iteration) AND have n eval rollout workers take some shot at an actual env? Just for research/debugging reasons.

What I'm saying is that we might want to consider separating the concepts of RolloutWorker/WorkerSet entirely from the concept of a separate dataset reader for offline data intake. Just some things to think about for the future.

There could be a use-case like that. Right now we don't support it. If we want to support it in the future we should create a separate project and re-define the evaluation config for completely isolating the two (or even more) eval flows.

Do we even support accepting both today? afaik the input config is like the following:

# for offline { "input": "dataset", "input_config": {"format": xxx, "path": [path1, path2]} } # for online { "input": "sampler", "input_config": {"env": "CartPole-v0"} }

Yeah, let's consolidate the entire input config API. I feel like there should be a single config method: input_setup or whatever name, which covers both our current environment() method as well as rollouts() (and offline_data() 😵‍💫 !).

To answer your question: Yeah, we have the mixed input, but it's not well explored/tested/configurable:

>>> MixedInput({ ... "sampler": 0.4, ... "/tmp/experiences/*.json": 0.4, ... "s3://bucket/expert.json": 0.2, ... }, ioctx)

yeah this will fall back to the old json reader I think which we want to get rid of anyway.

sven1977 · 2022-11-09T10:54:23Z

rllib/algorithms/algorithm.py

@@ -616,7 +616,8 @@ def setup(self, config: AlgorithmConfig) -> None:

        # Evaluation WorkerSet setup.
        # User would like to setup a separate evaluation worker set.
-        if self.config.evaluation_num_workers > 0 or self.config.evaluation_interval:
+        # Note: We skipp workerset creation if we need to do offline evaluation
+        if not self.config.get("off_policy_estimation_methods") and (self.config.evaluation_num_workers > 0 or self.config.evaluation_interval) :


self.config.off_policy_estimation_methods

sven1977 · 2022-11-09T10:55:40Z

rllib/algorithms/algorithm.py

@@ -2734,6 +2746,12 @@ def _run_one_evaluation(
        Returns:
            The results dict from the evaluation call.
        """
+
+        if self.config["off_policy_estimation_methods"]:


self.config.off_policy_estimation_methods

On my point above:
We should validate in the config, whether the user is trying to kind of do both: off_policy_setimation PLUS configuring some evaluation_duration stuff. This way we can error out and say: For now, you can only do either one of these types of evaluation: off-policy w/ dataset OR via environment rollouts.

If off_policy_estimation is specified and the input is not a dataset we should raise an error.

sven1977 · 2022-11-09T11:00:07Z

rllib/algorithms/algorithm.py

@@ -2817,6 +2835,19 @@ def _run_one_training_iteration_and_evaluation_in_parallel(

        return results, train_iter_ctx

+
+    def _run_offline_evaluation(self, train_future = None):


sven1977 · 2022-11-09T11:01:14Z

rllib/offline/estimators/fqe_torch_model.py

@@ -37,7 +38,7 @@ def __init__(
        Args:
            policy: Policy to evaluate.
            gamma: Discount factor of the environment.
-            model: The ModelConfigDict for self.q_model, defaults to:
+            model_config: The ModelConfigDict for self.q_model, defaults to:


Nice cleanup.

sven1977 · 2022-11-09T11:02:53Z

rllib/offline/offline_evaluator.py

+        n_parallelism: int = os.cpu_count(),
+    ) -> Dict[str, Any]:
+
+        """Calculates the estmiate of the metrics based on the given offline dataset. 


Describe more here how this happens: We run once(?) through the entire dataset?

sven1977 · 2022-11-09T11:03:11Z

rllib/offline/offline_evalution_utils.py

+
+    Args:
+        batch: The batch to remove the time dimension from.
+    Returns:


add empty line

Not sure I understand: Do we assume [B x T x d] or [T x B x d], just [T x d], correct?

each row in the dataset it assumed to be [T, d], there are N rows defining N episodes. When you put them in batches you'll get [B, T, d]. If it's a bandit setting you want [B, d]. I'll add more docstring around this.

added more docstrings.

sven1977 · 2022-11-09T11:04:29Z

rllib/offline/offline_evalution_utils.py

+def remove_time_dim(batch: pd.DataFrame) -> pd.DataFrame:
+    """Removes the time dimension from the given sub-batch of the dataset.
+
+    RLlib assumes each record in the dataset is a single episode. 


This information shouldn't go here, b/c it has nothing to do with the utility function per-se here.

got it. no information leakage from caller :)

sven1977 · 2022-11-09T11:06:07Z

rllib/offline/offline_evalution_utils.py

+        SampleBatch.NEXT_OBS,
+        SampleBatch.DONES,
+    ]:
+        batch[k] = batch[k].apply(lambda x: x[0])


What if the time dim > 1? Then we lose data?

yes. This should be used only in bandits setting. I have to think about non-bandit setting to see what is the best way to consolidate both.

The solution is to apply this only in the bandit settings for now and fall back to the old slow setup in case of non-bandit. User can specify a bandit setting via setting ope_split_batch_by_episode=False.

sven1977 · 2022-11-09T11:06:11Z

rllib/offline/offline_evalution_utils.py

+    Returns:
+        The batch with the time dimension removed.
+    """
+    for k in [


I know it's probably fine for now, but what about possibly other fields? Nested structures?

I don't think nested structures are supported in our offline RL stack anyway. So the assumption right now is that you have one key obs that is the flattened version of your input, i.e. if you need one-hot encoding, etc. you should do it on your dataset beforehand.

I resolved some issue with the keys that don't exist in the batch. But I think that's a separate problem from what u were concerned with.

sven1977 · 2022-11-09T11:06:28Z

rllib/offline/offline_evalution_utils.py

+    policy_state,
+    estimator_class,
+):  
+    """Computes importance sampling weights for the given batch of samples."""


proper dosctring

sven1977 · 2022-11-09T11:07:23Z

rllib/offline/offline_evalution_utils.py

+from typing import Any, Dict, Type, TYPE_CHECKING
+import numpy as np
+
+from ray.rllib.utils.numpy import convert_to_numpy


sort imports

sven1977 · 2022-11-09T11:31:37Z

rllib/offline/estimators/doubly_robust.py

@@ -56,6 +65,7 @@ def __init__(
            epsilon_greedy: The probability by which we act acording to a fully random
                policy during deployment. With 1-epsilon_greedy we act
                according the target policy.
+            normalize_weights: Whether to normalize the importance sampling


More description of what this arg does.

sven1977 · 2022-11-09T11:34:57Z

rllib/offline/estimators/doubly_robust.py

+
+        dsize = dataset.count()
+        batch_size = max(dsize // n_parallelism, 1)
+        # step 1: clean the dataset and remove the time dimension for bandits


Should we really auto-infer here that this is used for Bandits only? I feel like the time-dim removal doesn't belong here and should be handled by the client of this class.

yeah it doesn't belong here. It now happens in algorithm.py only if the problem is offline bandits

sven1977 · 2022-11-09T11:35:28Z

rllib/offline/estimators/direct_method.py

+
+        dsize = dataset.count()
+        batch_size = max(dsize // n_parallelism, 1)
+        # step 1: clean the dataset and remove the time dimension for bandits


see my comment below on why we shouldn't do the time-dim removal here.

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

…sense anymore Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

…ay into remove-evalworkers-in-ope

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha · 2022-11-14T06:06:44Z

rllib/algorithms/ddpg/ddpg.py

@@ -270,12 +270,6 @@ def validate(self) -> None:
                f"Try setting config.rollouts(rollout_fragment_length={self.n_step})."
            )

-        if self.model["custom_model"]:


apparently we never called this validate() function method before when custom_model was provided.

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

…ith dataset + tune Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha · 2022-11-14T21:44:22Z

@gjoliver @sven1977 Can one of you tell me why this documentation test is failing? I don't understand where the error is ?

gjoliver · 2022-11-14T22:12:12Z

@gjoliver @sven1977 Can one of you tell me why this documentation test is failing? I don't understand where the error is ?

are those red warnings related? I can't see anything man.
ask @maxpumperla maybe?

kouroshHakha · 2022-11-14T22:26:24Z

are those red warnings related? I can't see anything man.

They could be related. But I don't understand the error message. I updated those docstrings tbh but not sure what the proper format is and what the complain means.

gjoliver

nothing major.

gjoliver · 2022-11-15T08:12:37Z

rllib/algorithms/algorithm.py

+            # Dataset should be in form of one episode per row. in case of bandits each
+            # row is just one time step. To make the computation more efficient later
+            # we remove the time dimension here.
+            parallelism = self.evaluation_config.evaluation_num_workers or 1


we do have a parallelism parameter in evaluation_config. we don't want to use it in place of evaluation_num_workers?
it reads a little weird because on 1 hand, we say we are not gonna create evaluation workers, on the other hand, evaluation_num_workers actually play a critical role here.

The current users (you know who :)) get parallelism by defining the number of evaluation workers. I would love to actually change that at some point. To avoid further confusion I am getting rid of parallelism so it is less confusing. If you look in validate I am overriding the parallelism in evaluation_config.

gjoliver · 2022-11-15T08:22:26Z

rllib/algorithms/algorithm.py

+        self.evaluation_dataset = None
+        if (
+            self.evaluation_config.off_policy_estimation_methods
+            and not self.evaluation_config.ope_split_batch_by_episode


I have lost tracked a little bit. what if there are off_policy_estimation_methods and ope_split_batch_by_episode is True? how is that mode handled?
kinda of feel like we need a single functoin def _setup_evaluation(self), and in there, a simple list of different online/offline evaluation types that we may have, like:

def _setup_evaluation(self): if ( self.evaluation_config.off_policy_estimation_methods and not self.evaluation_config.ope_split_batch_by_episode ): self._setup_offline_bandit_eval() # DS based offline bandit data eval. elif self.config.evaluation_num_workers > 0: assert self.config.evaluation_config.env, ... self._setup_evaluation_workers() # Online eval. else: # What do we do here?

if ope is specified but not split_batch_by_episode we fall back to the old behavior. i.e. use evaluation workers that each have a shard of the dataset. It is not clear how I could use ray dataset map_batches to do processing on an episode level. In the future we can do that consolidation but till then let's just go back to the old behavior.

ok. but how do I tell this is the behavior here?
like we are gonna get to this if statement of off_policy_estimation_methods and not ope_split_batch_by_episode, if it doesn't fit, where do we fall to? how do I know the evaluation workers are guaranteed to be there?
to be clear, I am not saying the behavior is bad, I am just saying maybe we can write the code in a way that makes this flow clearer. wdyt?

yep. Let me explain what happens:
if this condition is true, you'll set self.evaluation_dataset to a dataset otherwise it's None. In self.evaluate() if self.evaluation_dataset is None you fall back on to the old behavior. which is using evaluation_workers. Now the question is have we actually created evaluation_workers when they where necessary? The answer is yes. It's in the setup() function where you call self._should_create_evaluation_rollout_workers(self.evaluation_config). How do you think this should be re-written so that it is more clear.

ok, fine for now.
personally I will feel a lot safer, if this whole thing is a single if...else..., then I don't all cases are covered.
right now creation of self.evalution_workers is conditioned on self._should_create_evaluation_rollout_workers(), and creation of evaluation_dataset is conditioned on off_policy_estimation_methods and not ope_split_batch_by_episode.
it's just hard to tell if these 2 are mutually exclusive or not. you get the idea.

gjoliver · 2022-11-15T08:23:58Z

rllib/algorithms/algorithm.py

@@ -803,6 +825,11 @@ def evaluate(
        # Call the `_before_evaluate` hook.
        self._before_evaluate()

+        if self.evaluation_dataset is not None:
+            eval_results = {}
+            eval_results["evaluation"] = self._run_offline_evaluation()


is this basically:

return { "evaluation": self._run_offline_evaluation() }

?

yes. fixed.

gjoliver · 2022-11-15T08:25:32Z

rllib/algorithms/algorithm.py

+            # training on the offline dataset, since rollout workers have already
+            # claimed it.
+            # Another Note (Kourosh): dataset reader will not use placement groups so
+            # what ever we specify here won't matter because dataset won't even use it.


gjoliver · 2022-11-15T08:28:59Z

rllib/algorithms/algorithm_config.py

+                # evaluation_num_workers for backward compatibility and num_cpus gets
+                # set to num_cpus_per_worker from rollout worker. User only needs to
+                # set evaluation_num_workers.
+                self.input_config["parallelism"] = self.evaluation_num_workers or 1


ok, so we should simply use "parallelism" in our code?

in code yes. from user it's evaluation_num_workers.

gjoliver · 2022-11-15T08:43:49Z

rllib/offline/estimators/tests/test_ope.py

+        check(is_ste, ope_results["is"]["v_gain_ste"])
+
+    def test_dr_on_estimate_on_dataset(self):
+        # TODO (Kourosh): How can we unittest this without querying into the model?


what do you mean? we can do hacky stuff in a unit test.

I guess my point was that when u do this you don't have a ground truth to compare against. We need to write this test in a way where the ground truth is independent of the implementation details. I don't know how to do that yet.

gjoliver · 2022-11-15T08:44:53Z

rllib/offline/resource.py

+        cpus_per_task = input_config.get(
+            "num_cpus_per_read_task", DEFAULT_NUM_CPUS_PER_TASK
+        )
+        return [{"CPU": cpus_per_task} for _ in range(parallelism)]


gjoliver · 2022-11-15T08:46:10Z

rllib/offline/feature_importance.py

+
+        For each feature in the dataset, the importance is computed by applying
+        perturbations to each feature and computing the difference between the
+        perturbed prediction and the reference prediction. The importance is


typo: is at the end is not necessary.

gjoliver · 2022-11-15T08:51:06Z

rllib/offline/feature_importance.py

+
+    Args:
+        dataset: The dataset to use for the computation. The dataset should have `obs`
+            and `actions` columns. Each record should be flat d-dimensional array.


mention that we only perturb / study features in the last dimension?

we compute it on all dims. I don't understand this comment.

line 253 n_features = ds.take(1)[0][SampleBatch.OBS].shape[-1]?

oh that's just getting the number of features.

but that's how many features we study right? when we perturb, we only shuffle along the last dim?

all of them. for each feature dim, you need to shuffle that entire column in the dataset. and do batch inference.

gjoliver · 2022-11-15T08:52:18Z

rllib/offline/feature_importance.py

+        The modified dataset that contains a `delta` column which is the absolute
+        difference between the expected output and the output due to the perturbation.
+    """
+    perturbed_ds = dataset.map_batches(


going over a same ds in parallel? this actually works? amazing.

actually good point. I have to verify this. I forgot to update the unittest for feature_importance. added a todo for now. We should get this merged and I'll add the unittest in a separate PR after the release timeline.

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

gjoliver

looks good now. I feel like the way we do offline evaluation now is finally right :)

Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: Weichen Xu <[email protected]>

kouroshHakha added 5 commits November 8, 2022 11:44

wip: working on wis

04e3838

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

wip: added is

3dd5c17

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

wip: parallelized feature_importance using ray core and added estimat…

678ab3a

…e_on_dataset Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

updated DR

e283531

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

wip added dm

550c0eb

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha assigned sven1977 and gjoliver Nov 9, 2022

kouroshHakha changed the title ~~[RLlib] Remove Evaluation workers when doing OPE~~ [RLlib] By-pass Evaluation workers when doing OPE Nov 9, 2022

gjoliver reviewed Nov 9, 2022

View reviewed changes

sven1977 reviewed Nov 9, 2022

View reviewed changes

kouroshHakha added 5 commits November 9, 2022 17:18

lint and resource allocation

732aeb9

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

parallel to trianing done

6b5ebde

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

fixing some unittests

24b8c53

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

fixed marwil tests

7c36c71

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

fixed unittest

c61a49d

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha added 5 commits November 13, 2022 21:37

remove some old mysterious ddpg validation check which does not make …

e0ebf5f

…sense anymore Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

Merge branch 'master' into remove-evalworkers-in-ope

877f7b2

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

Merge branch 'master' into remove-evalworkers-in-ope

18db127

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

Merge branch 'remove-evalworkers-in-ope' of github.com:kouroshHakha/r…

194fde8

…ay into remove-evalworkers-in-ope

removed resource pre-allocation for ray dataset per each tune trial

275be35

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha commented Nov 14, 2022

View reviewed changes

kouroshHakha added 2 commits November 13, 2022 22:24

fixed some previous review points

0b7bbd8

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

lint

e72e8c4

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha marked this pull request as ready for review November 14, 2022 16:53

kouroshHakha requested review from avnishn, ArturNiederfahrenhorst, smorad, maxpumperla and krfricke as code owners November 14, 2022 16:53

kouroshHakha added 4 commits November 14, 2022 10:44

added ste

4828f03

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

lint

72e2bd6

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

added some more comments to explain the resource allocation problem w…

7d93827

…ith dataset + tune Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

fixed ci

8641256

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

gjoliver reviewed Nov 15, 2022

View reviewed changes

kouroshHakha added 3 commits November 15, 2022 12:09

fixed doc

5fbb9c8

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

addressed jun's comments

43bacc4

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

raise value errors

eb82c84

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

gjoliver added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Nov 16, 2022

gjoliver approved these changes Nov 16, 2022

View reviewed changes

gjoliver merged commit e368dd9 into ray-project:master Nov 16, 2022

WeichenXu123 pushed a commit to WeichenXu123/ray that referenced this pull request Dec 19, 2022

[RLlib] By-pass Evaluation workers when doing OPE (ray-project#30135)

6a11700

Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: Weichen Xu <[email protected]>

		@@ -2817,6 +2835,19 @@ def _run_one_training_iteration_and_evaluation_in_parallel(

		return results, train_iter_ctx


		def _run_offline_evaluation(self, train_future = None):

[RLlib] By-pass Evaluation workers when doing OPE #30135

[RLlib] By-pass Evaluation workers when doing OPE #30135

Conversation

kouroshHakha commented Nov 9, 2022

Why are these changes needed?

Related issue number

Checks

kouroshHakha commented Nov 9, 2022

gjoliver left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sven1977 Nov 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kouroshHakha commented Nov 14, 2022

gjoliver commented Nov 14, 2022

kouroshHakha commented Nov 14, 2022

gjoliver left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sven1977 Nov 9, 2022 •

edited

Loading