Implement a Docker image generator gradle plugin for Spark applications #381

mccheah · 2018-06-09T00:08:26Z

Allows specifying the jars for the application's classpath as well as a custom base image.

mccheah · 2018-06-09T00:24:17Z

dev/publish_functions.sh

@@ -22,6 +22,9 @@ publish_artifacts() {
  echo "</server></servers></settings>" >> $tmp_settings

  ./build/mvn -T 1C --settings $tmp_settings -DskipTests "${PALANTIR_FLAGS[@]}" deploy
+  cd resource-managers/kubernetes/docker


We might want this to be done independently, e.g. I want to be able to retry only this part of the publish and not rerun the rest of the publish.

Do you actually need to cd here (would be less error prone if you don't need to)?

Yes we need to cd (because the git root isn't a gradle project, whereas resource-managers/kubernetes/docker is), and also I'd love to do this independently @mcheah.

Yep, see https://github.com/palantir/spark/pull/381/files/5a847bbc318d99e5c89e5e61a07fc7b2f2ce1dbc#r196244780

…publish-scripts

robert3005 · 2018-06-18T16:41:29Z

.circleci/config.yml

@@ -479,6 +495,10 @@ workflows:
          requires:
            - build-sbt
          <<: *all-branches-and-tags
+      - run-spark-docker-gradle-plugin-tests:


you need to wire publishing as well

Should be included because we put the gradle publish call in publish_functions.sh

robert3005 · 2018-06-18T16:42:55Z

dev/build-spark-docker-gradle-plugin.sh

+set -euo pipefail
+ROOT=$(git rev-parse --show-toplevel)
+cd "$ROOT/resource-managers/kubernetes/docker"
+./gradlew --info compileJava compileTestJava


ideally you can just call resource-managers/kubernetes/docker/gradlew?

why not build?

Re calling resource-managers/kubernetes/docker/gradlew, you have to be in the gradle build's root project, otherwise you'd need to set it with -p.

build runs the tests too which is redundant given that we run tests in a separate phase. But it's probably fine to just have one build step that runs both compile and test.

Regarding running without changing directories, you get this:

FAILURE: Build failed with an exception. * What went wrong: Task 'compileJava' not found in root project 'spark'.

Gradle infers the project from the current working directory.

Can resolve with https://stackoverflow.com/questions/15374409/how-to-pass-build-gradle-directory-to-gradlew

robert3005 · 2018-06-18T16:53:26Z

@mccheah looks good - mostly would like to figure out if we can remove some boilerplate gradle scripts

robert3005 · 2018-06-18T16:54:00Z

resource-managers/kubernetes/docker/settings.gradle

+ * limitations under the License.
+ */
+
+rootProject.name = 'spark-docker-image-generator_2.11'


don't add 2.11 - this isn't a scala project?

robert3005 · 2018-06-18T16:56:31Z

...docker/src/main/java/org/apache/spark/deploy/kubernetes/docker/gradle/DeployScriptsTask.java

+             new TarArchiveInputStream(dockerResourcesGZipped)) {
+      TarArchiveEntry nextEntry = dockerResourcesTar.getNextTarEntry();
+      while (nextEntry != null) {
+        boolean extracted = maybeCopyTarEntry(dockerResourcesTar,


can we extract once and copy?

This whole function can be described using a Sync task (or, if we need dockerBuildDir to be defined lazily, then can use an inline action project.sync(spec -> ...).
Picking selective files from a tar is easy, though there is a bit more logic necessary if you wanna rename paths... as an example to get started:

// Not 100% sure if you can give it a URL but from looking into the code it should work // Alternatively you can implement a `ReadableResource` and pass that to tarTree FileTree tree = project.tarTree(getClass().getResource("/docker-resources.tgz")); project.sync(spec -> { spec.from(tree, s2 -> { s2.include("docker-resources/bin", "docker-resources/sbin"); }); // TODO implement stripFirstDirectoryFrom - in groovy you'd do { it.relativePath.segments.toList()[1..-1].join("/") } spec.eachFile(fcd -> fcd.setRelativePath(stripFirstDirectoryFrom(fcd.getRelativePath()))); spec.into(outputDir); });

Then you don't need deleteAndMkDirs either, Sync automatcally copies just what was included by the from blocks into the into directory, removing everything else that's not supposed to be there.

Already had to implement this somewhere else and can confirm it works. Implementation for stripFirstDirectoryFrom:

RelativePath stripFirstDirectoryFrom(RelativePath relativePath) { String[] segments = relativePath.getSegments(); return new RelativePath(relativePath.isFile(), Arrays.copyOfRange(segments, 1, segments.length)); }

We actually don't want to extract the tgz with just the top level directory removed - we want to also be dropping the files in specific locations so that the docker build can work with the Dockerfile properly when adding files. Thus something like this:

else if (sourcePathSegments[1].equals("entrypoint")) { copyDetails.setRelativePath( RelativePath.parse( true, "kubernetes/dockerfiles/spark/" + withoutTopLevel)); }

The alternative is building the tgz differently such that it's easier to extract on the task execution side.

robert3005 · 2018-06-18T17:11:52Z

...r/src/main/java/org/apache/spark/deploy/kubernetes/docker/gradle/GenerateDockerFileTask.java

+                        return String.format("FROM %s", getBaseImage());
+                      } else if (line.equals(
+                          "    apk add --no-cache bash tini libc6-compat && \\")) {
+                        return line.replace("libc6-compat ", "");


why not just remove this from image?

Good question, we'll discuss this in the community - probably we want custom image authors to be providing this component themselves. For now I think it's fine to fork the Dockerfile.

Wait - this is our fork, let's just remove it from dockerfile here

robert3005 · 2018-06-18T17:13:09Z

...r/src/main/java/org/apache/spark/deploy/kubernetes/docker/gradle/GenerateDockerFileTask.java

+               BufferedReader sourceDockerFileBuffered =
+                   new BufferedReader(sourceDockerFileReader)) {
+            List<String> fileLines = Collections.unmodifiableList(
+                sourceDockerFileBuffered.lines()


Files.lines()? Just remember to close the stream

The source docker file here is embedded inside the tarball. You can't get a Path or a File object directly from the archive entry, at least not one which can be used to open a stream.

How about not embedding a tgz? Not sure how many files are actually there. Then you could use https://google.github.io/guava/releases/21.0/api/docs/com/google/common/io/Resources.html#readLines-java.net.URL-java.nio.charset.Charset-

robert3005 · 2018-06-18T17:31:57Z

...docker/src/main/java/org/apache/spark/deploy/kubernetes/docker/gradle/SparkDockerPlugin.java

+          + dockerBuildDirectory.getAbsolutePath());
+    }
+
+    project.afterEvaluate(evaluatedProject -> {


This shouldn't be using afterEvaluate - dependencies should be setup via task inputs and outputs

In sync. If any of the task inputs need to be computed lazily, declare a property field in the task which you can create with project.getObjects().property<...>(), and instantiate it with a provider like task.theProperty.set(project.provider(() -> ...))

Have a look at https://docs.gradle.org/current/userguide/lazy_configuration.html which describes how to use these in more detail

Some of the wiring here relies on the jar task being set up already -n are we saying that the plugin user should be passing that output jar in specifically? I'm not sure how we get the output of the jar task injected into the image without using afterEvaluate to verify that the jar task exists in the first place.

The jar task comes from JavaPlugin, you can do smth like

project.getPluginManager().withPlugin("java", javaPlugin -> { // access the jar task here, knowing it has been created });

dansanduleac · 2018-06-18T17:42:25Z

dev/build-spark-docker-gradle-plugin.sh

+set -euo pipefail
+ROOT=$(git rev-parse --show-toplevel)
+cd "$ROOT/resource-managers/kubernetes/docker"
+./gradlew --info compileJava compileTestJava


Re calling resource-managers/kubernetes/docker/gradlew, you have to be in the gradle build's root project, otherwise you'd need to set it with -p.

dansanduleac · 2018-06-18T17:43:49Z

dev/publish_functions.sh

@@ -22,6 +22,9 @@ publish_artifacts() {
  echo "</server></servers></settings>" >> $tmp_settings

  ./build/mvn -T 1C --settings $tmp_settings -DskipTests "${PALANTIR_FLAGS[@]}" deploy
+  cd resource-managers/kubernetes/docker


Yes we need to cd (because the git root isn't a gradle project, whereas resource-managers/kubernetes/docker is), and also I'd love to do this independently @mcheah.

dansanduleac · 2018-06-18T17:46:02Z

dev/run-spark-docker-gradle-plugin-tests.sh

+set -euo pipefail
+ROOT=$(git rev-parse --show-toplevel)
+cd "$ROOT/resource-managers/kubernetes/docker"
+./gradlew --info test


Why do these scripts just assume they'll be executing in kubernetes/docker? I'd rather be explicit about this in circle, something like ./build/gradlew -p resource-managers/kubernetes/docker --info test

dansanduleac · 2018-06-18T17:49:24Z

resource-managers/kubernetes/docker/build.gradle

+
+task prepareResources(type:Exec) {
+  workingDir "$rootDir"
+  commandLine 'scripts/copy-docker-items-to-resources.sh'


This will always execute and can't take advantage of caching.
Looking at the script, it feels like this can be accomplished with two gradle tasks:

Sync a bunch of stuff into DOCKER_BUNDLE_DIR. There are a bunch of examples in this working with files document about how to create file trees, use them as inputs, filter them etc.

Tar (see creating archives, but use Tar instead of Zip & can enable compression too) that directory into an archive. There is still one gotcha here that you can only

Note: MODULE_ROOT_DIR is just (project.)rootDir etc and I'd recommend creating outputs under the build directory as is customary/default in gradle, as opposed to target.
So DOCKER_BUNDLE_DIR could also just be file("${buildDir}/docker-resources")

dansanduleac · 2018-06-18T17:53:41Z

...docker/src/main/java/org/apache/spark/deploy/kubernetes/docker/gradle/SparkDockerPlugin.java

+
+  @Override
+  public void apply(Project project) {
+    project.getPluginManager().apply(getClass());


Don't do this - this is how to apply another plugin to a project. This plugin is already being applied by virtue of being in this method.

dansanduleac · 2018-06-18T18:06:33Z

...docker/src/main/java/org/apache/spark/deploy/kubernetes/docker/gradle/SparkDockerPlugin.java

+        "-t",
+        extension.getImageName(),
+        buildDirectory.getAbsolutePath());
+    dockerBuild.setDependsOn(


dependsOn(...)

dansanduleac · 2018-06-18T18:08:53Z

...docker/src/main/java/org/apache/spark/deploy/kubernetes/docker/gradle/SparkDockerPlugin.java

+        extension.getImageName(),
+        buildDirectory.getAbsolutePath());
+    dockerBuild.setDependsOn(
+        Arrays.asList(evaluatedProject.getTasks().getByName("sparkDockerPrepare")));


can give it just strings - dependsOn("sparkDockerPrepare")

dansanduleac · 2018-06-18T22:24:42Z

...docker/src/main/java/org/apache/spark/deploy/kubernetes/docker/gradle/SparkDockerPlugin.java

+              "docker",
+              "tag",
+              extension.getImageName(),
+              String.format("%s:%s", extension.getImageName(), tag));


Sadly the Exec task doesn't readily accept lazy arguments, so I can see why this wouldn't work unless you did afterEvaluate.
The better way to do it though would be to create a custom class extending DefaultTask that declares Property<...> for any lazy arguments it needs, and do the work manually in a @TaskAction-annotated method (can make use of project.exec(spec -> ...) where things like setCommandLine would be called on the spec.

dansanduleac · 2018-06-19T15:06:00Z

...docker/src/main/java/org/apache/spark/deploy/kubernetes/docker/gradle/DeployScriptsTask.java

+             new TarArchiveInputStream(dockerResourcesGZipped)) {
+      TarArchiveEntry nextEntry = dockerResourcesTar.getNextTarEntry();
+      while (nextEntry != null) {
+        boolean extracted = maybeCopyTarEntry(dockerResourcesTar,


This whole function can be described using a Sync task (or, if we need dockerBuildDir to be defined lazily, then can use an inline action project.sync(spec -> ...).
Picking selective files from a tar is easy, though there is a bit more logic necessary if you wanna rename paths... as an example to get started:

// Not 100% sure if you can give it a URL but from looking into the code it should work // Alternatively you can implement a `ReadableResource` and pass that to tarTree FileTree tree = project.tarTree(getClass().getResource("/docker-resources.tgz")); project.sync(spec -> { spec.from(tree, s2 -> { s2.include("docker-resources/bin", "docker-resources/sbin"); }); // TODO implement stripFirstDirectoryFrom - in groovy you'd do { it.relativePath.segments.toList()[1..-1].join("/") } spec.eachFile(fcd -> fcd.setRelativePath(stripFirstDirectoryFrom(fcd.getRelativePath()))); spec.into(outputDir); });

dansanduleac · 2018-06-19T15:07:16Z

...docker/src/main/java/org/apache/spark/deploy/kubernetes/docker/gradle/DeployScriptsTask.java

+             new TarArchiveInputStream(dockerResourcesGZipped)) {
+      TarArchiveEntry nextEntry = dockerResourcesTar.getNextTarEntry();
+      while (nextEntry != null) {
+        boolean extracted = maybeCopyTarEntry(dockerResourcesTar,


Then you don't need deleteAndMkDirs either, Sync automatcally copies just what was included by the from blocks into the into directory, removing everything else that's not supposed to be there.

mccheah · 2018-06-19T18:36:22Z

@dansanduleac do you have any advice for how to create tasks given the setup of the extension, without using the afterEvaluate method?

I looked at https://github.com/palantir/gradle-docker/blob/develop/src/main/groovy/com/palantir/gradle/docker/PalantirDockerPlugin.groovy#L117 and in particular it generated a separate docker tag task for every tag to be generated. But it had to use afterEvaluate to do that, and I'm wondering if we can do something more idiomatic.

Taking a step back though I wouldn't be entirely opposed to the idea of only allowing for a single tag per Spark application image, so if accomplishing this would require an overly complex setup we can make this only assign one tag.

mccheah · 2018-06-19T23:37:53Z

Hm, we could define a task Rule which checks if the extension defined the given tag and constructs the task on the fly. Would that work? https://www.oreilly.com/library/view/gradle-beyond-the/9781449373801/ch03.html. I'll give this a try.

mccheah · 2018-06-20T02:07:32Z

Actually Rule and Model based Gradle development is a pretty significant paradigm shift altogether. It might be correct for this situation though. Will do some experimentation with the APIs.

dansanduleac · 2018-06-22T11:34:08Z

Yep if you want to create a different number of tasks based on runtime values, then you need afterEvaluate (and wire them up via dependencies to a task that you explicitly create outside of afterEvaluate). But perhaps you can do only that bit in afterEvaluate, that would simplify the flow a lot.
Not sure if it's worth complicating matters with the Rule api which seems to me as powerful as scala implicits but with the added blackboxiness of doing everything via reflection.

…publish-scripts

mccheah · 2018-06-25T23:25:10Z

@dansanduleac @robert3005 think I've addressed all the comments. Will add some more tests but more feedback is appreciated.

robert3005 · 2018-06-26T23:28:56Z

@mccheah how about instead of putting docker files as a tgz in the jar we publish them separately? This would be akin to how gradle-baseline does things where configuration is an input to the plugin. This would let us remove some of the weird file reading and in general be more gradle friendly

robert3005 · 2018-08-16T01:00:26Z

...ker/src/test/java/org/apache/spark/deploy/kubernetes/docker/gradle/DockerBuildTaskSuite.java

+        dockerBuildDirectory.getAbsolutePath());
+  }
+
+}


missing newline character

robert3005 · 2018-08-16T01:01:24Z

...ker/src/test/java/org/apache/spark/deploy/kubernetes/docker/gradle/DockerBuildTaskSuite.java

+import java.io.File;
+import java.io.IOException;
+
+public final class DockerBuildTaskSuite {


prefer use of mockitojunitrunner since initMocks no longer works in junit5 and it's more obvious to migrate since the runner no longer exists there

robert3005

Still there's weirdness here - you shouldn't need after evaluate. You want to string everything together with providers/properties and leave getting values to gradle

mccheah · 2018-08-16T01:10:02Z

Agreed that properties should be used in the extension. We can move the project up to the root directory, I mostly didn't want to make the root directory too different from stock Spark's (i.e. isolate the added code to be in the resource-managers/kubernetes/docker subdirectory). The use of afterEvaluate as well as custom exec tasks is a necessary evil - please refer to earlier discussion in the PR.

robert3005 · 2018-08-16T02:57:47Z

Makes sense - one more question though. Why do we need tags? They're not useful and it's easier to just publish snapshot and release versions. Internally we stopped relying on latest and snapshot long time ago and I don't see a good reason why you'd want them.

mccheah · 2018-08-16T20:35:25Z

The cue is mainly taken from gradle-docker: https://github.com/palantir/gradle-docker/blob/develop/src/main/groovy/com/palantir/gradle/docker/DockerExtension.groovy#L35. But this is more by convention and consistency with that plugin. In practice tag should more or less be the git tag, or the git hash if it's a snapshot..

- Move Gradle project to the root directory - Use Gradle 4.9 - More properties - Use baseline

mccheah · 2018-08-18T00:43:16Z

.baseline/checkstyle/checkstyle-suppressions.xml

+
+    <!-- Generated code should not be subjected to checkstyle. -->
+    <suppress files="[/\\].*[/\\]generated.*[/\\]" checks="." />
+    <suppress files="[/\\].*[/\\]docker-resources.*[/\\]" checks="." />


Avoid invoking checkstyle on shell scripts that are placed in the resources folder

mccheah · 2018-08-18T00:43:46Z

.baseline/checkstyle/checkstyle.xml

+    </module>
+    <module name="Header">
+        <!--<property name="header" value="^/\*$\n^ \* Licensed to the Apache Software Foundation (ASF) under one or more$\n"/>-->
+        <property name="headerFile" value="${config_loc}/header.txt"/>


Make the header use only the Apache license

mccheah · 2018-08-18T00:44:05Z

.baseline/checkstyle/checkstyle.xml

+        <module name="Indentation"> <!-- Java Style Guide: Block indentation: +2 spaces -->
+            <property name="arrayInitIndent" value="4"/>
+            <property name="lineWrappingIndentation" value="4"/>
+            <property name="basicOffset" value="2"/>


All indentation is 2 spaces to be consistent with the rest of the Spark codebase

mccheah · 2018-08-18T00:45:17Z

.baseline/checkstyle/checkstyle.xml

+            <message key="illegal.type" value="Do not use Guava caches, they are outperformed by and harder to use than Caffeine caches"/>
+        </module>
+        <!-- Use Spark's built-in import ordering -->
+        <module name="ImportOrder">


Import ordering matches the rest of Spark, although from what we can tell this hasn't been enforced consistently. https://spark.apache.org/contributing.html

mccheah · 2018-08-18T00:46:01Z

dev/.rat-excludes

@@ -113,3 +113,8 @@ structured-streaming/*
 kafka-source-initial-offset-version-2.1.0.bin
 kafka-source-initial-offset-future-version.bin
 vote.tmpl
+gradle-build/*


Avoid inspecting files built by Gradle for license checks

mccheah · 2018-08-18T00:51:01Z

@robert3005 addressed comments. I've noted the differences to stock baseline in comments here.

if you feel strongly about not supporting multiple tags then we can remove it.

mccheah · 2018-08-18T00:52:00Z

build.gradle

+allprojects {
+  group 'org.apache.spark'
+  version System.env.CIRCLE_TAG ?: gitVersion()
+  buildDir = 'gradle-build'


Because Spark's root directory already contains a folder called build, presumably for sbt, and we can't delete its contents freely with ./gradlew clean.

dansanduleac

had some comments from before

dansanduleac · 2018-07-19T19:17:58Z

resource-managers/kubernetes/docker/build.gradle

+  testCompile gradleTestKit()
+}
+
+def getGitRepoRoot = { ->


this is a closure whether you add -> or not, can remove it

This function isn't in the latest version.

dansanduleac · 2018-07-20T10:53:50Z

resource-managers/kubernetes/docker/build.gradle

+    into 'kubernetes/dockerfiles/spark'
+  }
+
+  from(fileTree("$getGitRepoRoot/bin")) {


can we just use a relative path here

I believe this doesn't apply anymore, given how we've rearranged the Gradle project.

We instead use rootDir instead of getGitRepoRoot. if we were to use relative paths, we'd have to refer to the parent path, e.g. from(fileTree("../bin")). I'm generally against referencing the parent path this way because it requires one to know where the script is running from. Referencing the absolute path gives us a path that's consistent regardless of where the script is running from or if we modify the directory structure of this project later.

robert3005 · 2018-08-23T14:42:20Z

dev/run-spark-docker-gradle-plugin-tests.sh

+# limitations under the License.
+#
+set -euo pipefail
+./gradlew --info --stacktrace test


we shouldn't need this - just call the gradle command, you gain nothing by wrapping it

robert3005 · 2018-08-23T14:43:51Z

@mccheah lets remove the wrapping scripts after that it's LGTM from me

robert3005 · 2018-08-30T10:26:01Z

...erator/src/main/java/org/apache/spark/deploy/kubernetes/docker/gradle/SparkDockerPlugin.java

+          commandLine.add(extension.getImageName());
+          task.setCommandLine(commandLine);
+        });
+    project.afterEvaluate(evaluatedProject -> {


I think this is not necessary as long as we make configuration of the extension lazy. If you look at how MavenPublish plugin works from 4.8 onwards is that it creates tasks in all action on publication container. Hence, we can assume that any time you'd need an after evaluate is an error since it means the user force project evaluation at some point.

Simple thing to do here would be to make tags NamedDomainObjectContainer and then you can lazily register things by name which would be the tag.

This can be done separately as long as it's done

mccheah · 2018-08-31T00:55:39Z

@dansanduleac @robert3005 going to merge after latest CI passes if there are no more comments.

robert3005 · 2018-08-31T01:09:39Z

Cool with me, just remember to squash

mccheah · 2018-08-31T18:19:29Z

Merge is complete, thanks!

mccheah · 2018-08-31T18:26:34Z

Whoops it broke master - will fix

mccheah added 5 commits June 8, 2018 16:57

Implement a Docker image generator gradle plugin for Spark applications.

42988ef

Fix circle

4b32aa9

Fix more circle

6bded05

No need to setup docker for compilation only

2bb1702

Fix gradle

cad9d41

mccheah commented Jun 9, 2018

View reviewed changes

mccheah added 2 commits June 14, 2018 16:30

Add license headers

92139c0

Merge remote-tracking branch 'palantir/master' into add-docker-image-…

5a847bb

…publish-scripts

robert3005 reviewed Jun 18, 2018

View reviewed changes

dansanduleac suggested changes Jun 19, 2018

View reviewed changes

mccheah added 6 commits June 25, 2018 13:41

Address comments.

46f23c9

Merge remote-tracking branch 'palantir/master' into add-docker-image-…

71b835a

…publish-scripts

Remove extra task

4149dfd

Remove 2.11

814978f

Remove extra script

e123548

Remove some imports

6bf957a

Add more tests

3194cd1

robert3005 reviewed Aug 16, 2018

View reviewed changes

robert3005 suggested changes Aug 16, 2018

View reviewed changes

mccheah added 4 commits August 17, 2018 14:25

Address comments.

999b5f8

- Move Gradle project to the root directory - Use Gradle 4.9 - More properties - Use baseline

Fix build script

6704dd5

Fix build again

e2c862d

Fix licenses and ignore build dir licenses

3017341

mccheah commented Aug 18, 2018

View reviewed changes

dansanduleac reviewed Aug 22, 2018

View reviewed changes

robert3005 reviewed Aug 23, 2018

View reviewed changes

robert3005 reviewed Aug 30, 2018

View reviewed changes

Remove extraneous scripts

fb57b7e

mccheah merged commit 036259f into master Aug 31, 2018

mccheah deleted the add-docker-image-publish-scripts branch August 31, 2018 18:19

jdcasale mentioned this pull request Feb 19, 2021

Spark3 diff on top of tag 3.0.1 #735

Closed

Implement a Docker image generator gradle plugin for Spark applications #381

Implement a Docker image generator gradle plugin for Spark applications #381

Conversation

mccheah commented Jun 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robert3005 commented Jun 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dansanduleac Jun 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dansanduleac Jun 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mccheah commented Jun 19, 2018

mccheah commented Jun 19, 2018 • edited Loading

mccheah commented Jun 20, 2018

dansanduleac commented Jun 22, 2018

mccheah commented Jun 25, 2018

robert3005 commented Jun 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robert3005 left a comment

Choose a reason for hiding this comment

mccheah commented Aug 16, 2018

robert3005 commented Aug 16, 2018

mccheah commented Aug 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mccheah commented Aug 18, 2018

Choose a reason for hiding this comment

dansanduleac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robert3005 commented Aug 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mccheah commented Aug 31, 2018

robert3005 commented Aug 31, 2018

mccheah commented Aug 31, 2018

mccheah commented Aug 31, 2018

dansanduleac Jun 19, 2018 •

edited

Loading

dansanduleac Jun 19, 2018 •

edited

Loading

mccheah commented Jun 19, 2018 •

edited

Loading