Clojure Contrib Package #11205

gigasquid · 2018-06-08T18:18:43Z

Description

This adds a Clojure package for MXNet under a contrib package

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

[ X] Changes are complete (i.e. I finished coding on this PR)
[ X] All changes have test coverage:
[ X] Code is well-documented:
[ X] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

This is the initial commit of the Clojure package in as a contrib project. There will be a period of stabilization, feedback, and contribution. After that, it may graduate to out of contrib.

The initial package has been tested by the community in a separate repo prior to this PR https://github.com/gigasquid/clojure-mxnet

Here is the original issue:
#8971

Architecture & Design:
https://cwiki.apache.org/confluence/display/MXNET/MXNet+Clojure

Current List of Package Completion and Contribution Needs
https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs

Comments

There have been areas of improvement noted from feedback on the mailing list.

Increased test coverage
Benchmarks

In addition there will be a need to integrate with the Makefile and CI.

I would like to propose that this initial PR be merged as it represents a working, community tested development milestone and then add on the improvements as separate PRs to allow smaller commits, better visibility, and community involvement.

gigasquid · 2018-06-08T20:00:15Z

@nswamy This is ready for review/feedback

lanking520 · 2018-06-08T21:35:54Z

~~Hi Carin, is there anyway I can test it? I am pretty new to Clojure~~
Found tutorial in the PR, my bad, will test it

kurman-zz · 2018-06-08T21:50:57Z

Do you think you can split examples from cde0ee3 into a separate commit to make the review process easier?

gigasquid · 2018-06-08T22:32:41Z

@kurman sure - I will rewrite the commits and and push tomorrow (wrapping up today 😄 )

gigasquid · 2018-06-09T15:33:40Z

@kurman - I rewrote the commits into 2 parts - one with the main package and tests and another with the examples. Please let me know if I can do anything else to make the review process easier.

Clean up dependencies in namespaces to make it compatible with the plugin

kurman-zz · 2018-06-12T07:47:48Z

@gigasquid thanks for splitting the commit, I am already finding that it is easer going through the PR.

kovasb · 2018-06-14T23:20:12Z

Thanks for this contribution!

In terms of code quality this looks very good, getting a lot done in a straightforward manner. People should feel good shipping this if/when test coverage is satisfactory.

My only code question is: Where do the corner cases tend to lie? Whats the most complicated part where bugs might be lurking?

Some higher level questions:

What is the debug experience like? It seems like errors can come from 3 places (mxnet core, scala, or clojure), is this a problem in practice? Can users track down what they did wrong?
What does maintenance look like? As MXNet & Scala API evolve, how much work does this codebase need to do to keep up? What changes are picked up automatically via introspection, vs requiring changing the codebase?
How do people learn to use the thing? Is it easy to go between the clojure functions and existing MXNet docs/tutorials? How is the documentation going to work long term?

For a v1 the answers to a lot of these are probably uncertain, but to the extent the code/design already has an opinion/constraint that might be useful to know.

gigasquid · 2018-06-15T12:43:55Z

@kovasb Thanks so much for taking the time to review and providing such insightful feedback and questions.

As far as corner cases go, I feel pretty confident about the interop with the Scala api. I would say the risk for complications would be lurking in the areas that I haven't fully ported over yet and explored. I ran into some trouble with the protocols for the custom operators with class loading and deferred working on it more to focus on some other areas. I'm hoping it was just some silly thing that I wasn't doing properly because in theory, if it's a Scala/Java class - it can all be interoped with, but I won't be 100% sure until I dive in again.

You have some excellent higher level questions:

The debug experience is an area that could use improvement. I've added it as an item on the https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs page. The regular JVM stack traces that occur in clojure point to the errors in the Scala or Clojure class and can be pinned down pretty readily and are not bad. The more frustrating ones that I have encountered is when there is a core dump, (for example binding some layers with a bad shape), in the mxnet-core. You can still find the line where the error occurred in the underlying Scala/Clojure code, but it is not easy to navigate. From what I understand, I believe that this might be the same case in the current Scala package as well, so I'm thinking that working together we could make it better for both packages if we could tackle this.
I think the story for maintenance is pretty good because of leveraging the Scala package. The Clojure package also has the flexibility to evolve at its own rate by deciding when to move to the next Scala version or wrap a new Scala api. I think the philosophy in MXNet is to not make breaking changes is very good and will enable stability. The mxnet core api changes for NDArray and Symbol flow through the Scala package via the c code generators. The clojure package picks these changes up from the Scala classes by running its own generator, (which is a simple lein command). Changes to the higher level Scala apis will also be picked up or course, while any new apis would need to be added and incorporated manually.
The Clojure apis have been developed with the goal of being intuitive from the existing apis while also being idiomatic to use from the Clojure perspective. I've been able to use existing tutorials from both Python and Scala and translate them to Clojure. Here is an example:

From the Python tutorial https://mxnet.incubator.apache.org/tutorials/basic/symbol.html you have:

net = mx.sym.Variable('data')
net = mx.sym.FullyConnected(data=net, name='fc1', num_hidden=128)
net = mx.sym.Activation(data=net, name='relu1', act_type="relu")
net = mx.sym.FullyConnected(data=net, name='fc2', num_hidden=10)
net = mx.sym.SoftmaxOutput(data=net, name='out')

Translation to clojure follows pretty directly from this:

(def data (sym/variable "data"))
(def fc1 (sym/fully-connected "fc1" {:data data :num-hidden 128}))
(def act1 (sym/activation "act1" {:data fc1 :act-type "relu"}))
(def fc2 (sym/fully-connected "fc2" {:data act1 :num-hidden 64}))
(def net (sym/softmax-output "out" {:data fc2}))

But also can be written in a more idiomatic, threading way using the as-> clojure function

(as-> (sym/variable "data") data
  (sym/fully-connected "fc1" {:data data :num-hidden 128})
  (sym/activation "act1" {:data data :act-type "relu"})
  (sym/fully-connected "fc2" {:data data :num-hidden 64})
  (sym/softmax-output "out" {:data data}))

As far as documentation goes, the design choice has been made to generate the code from the symbol and ndarray apis and persist that to files rather than read into memory as macros so that it enable automated API documentation with Codox. Going forward, I envision a section of the MXNet webpage with tutorials and API documentation similar to the other language bindings to also help the user getting started.

larroy · 2018-06-16T15:35:21Z

contrib/clojure-package/examples/cnn-text-classification/get_data.sh

+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+


Could you add "set -e" to this script?

isn’t that just below set -evx ?

larroy · 2018-06-16T15:38:02Z

contrib/clojure-package/README.md

+### Use Prebuilt Jars
+There are deployed jars on Clojars for each supported system
+
+* `[org.apache.clojure-mxnet/clojure-mxnet-linux-gpu "0.1.1-SNAPSHOT"]`


Who is building these packages? How is the version number related to the mxnet version?

Can we add instructions to build the package as well? Something like when you build a scala package in the local ivy repo.

That's a good point. The section was taken from the README in the external git repo that I had for testing. I had built and deployed those jars manually and pushed them to clojars.

I'm not sure how the build will happen within the project when it gets merged long term. We will need some discussion on that. For the short term, I can continue to build the jars manually and push them to clojars for general users to consume and give feedback on.

I definitely can document the process that I use to do the build and deploy 👍

larroy · 2018-06-16T15:41:05Z

contrib/clojure-package/examples/pre-trained-models/download-resnet-152.sh

+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+


larroy · 2018-06-16T15:41:12Z

contrib/clojure-package/examples/pre-trained-models/download-caltech.sh

+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+


larroy

Great contribution!

gigasquid · 2018-06-16T19:37:38Z

Thanks for taking the time to review @larroy 😸 I'll get to work on incorporating your feedback.

- Also fix parameter names to lowercase

larroy

Package testing LGTM except the OpenCV dependency. Can you comment on the OpenCV part?

larroy · 2018-06-27T01:03:09Z

ci/docker/install/ubuntu_clojure.sh

+chmod 777 lein
+sudo cp lein /usr/local/bin
+sudo apt install libcurl3
+sudo add-apt-repository ppa:timsc/opencv-3.4


Is the opencv version that is in ubuntu too old? Is not clear why we are adding this to me.

I think it's also causing some issues in CI:

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11205/19/pipeline

yes - that is the issue I'm looking into. It appears that the version of the opencv that was used to build the scala jars for nexus is different than the opencv required by caffe. I'm looking into changing the unit test from the clojure package to run off of the ones generated by make scalapkg and make scalainstall instead, (which is most likely a better way to go anyway).

Using the scala jars from nexus on ubuntu uses this version
sudo apt install -y libopencv-imgcodecs3.4
which clashes with the one required in caffe build libopencv-dev

…ge version -remove makefile modification and use script instead

gigasquid · 2018-06-27T12:38:06Z

@larroy The CI Clojure unit tests now pass 😸
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11205/22/pipeline

cjolivier01 · 2018-06-28T03:14:01Z

contrib/clojure-package/README.md

+
+## Introduction
+
+MXNet is a first class, modern deep learning library that AWS has officially picked as its chosen library. It supports multiple languages on a first class basis and is incubating as an Apache project.


should AWS be mentioned at all?

I'm a little confused by the +1 - I'm interpreting that it's ok to leave the AWS in the text, but if I understood that wrong, please let me know :)

No, please remove all mentions of AWS.

cjolivier01 · 2018-06-28T04:26:00Z

contrib/clojure-package/examples/cnn-text-classification/get_data.sh

+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+


isn’t that just below set -evx ?

marcoabreu

Made a rough pass and everything LGTM, thanks a lot everybody!

larroy · 2018-06-28T17:33:04Z

contrib/clojure-package/src/org/apache/clojure_mxnet/module.clj

+                                 (callback/invoke cb i batch-num eval-metric))
+                               (.dispose batch)
+                          (inc batch-num)))
+       (println "Epoch " i " Train-" (eval-metric/get eval-metric))


isn't it a better practice to use logging? in Scala we often use logback for logging, so you can configure logging for these classes independently.

Thanks - excellent point. I put an item to improve that in https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs

larroy

LGTM, I added a comment WRT println vs logging. But this should not block the PR. Amazing Job! thanks for the contribution!

lanking520

Thanks @gigasquid for your contribution and I can't wait to see Clojure go into MXNet 😊

lanking520 · 2018-06-28T20:39:06Z

contrib/clojure-package/README.md

+`cd ~/mxnet`
+
+
+`git checkout tags/1.2.0 -b release-1.2.0`


Does the package requires a fixed version of MXNet as 1.2 or it can be updated accordingly with mxnet?

lanking520 · 2018-06-28T20:40:38Z

contrib/clojure-package/README.md

+
+**Why build on the Scala package?**
+
+The motivation section addresses this, but the main reason is high leverage is using the great work that the Scala package has already done.


lanking520 · 2018-06-28T20:42:21Z

contrib/clojure-package/README.md

+
+The motivation section addresses this, but the main reason is high leverage is using the great work that the Scala package has already done.
+
+**How can I tell if the gpu is being used?**


It can also be shown in the message that CUDA is finding a best algorithm... As long as a Context.gpu() passed in the code as a context, GPU should be used

Thanks I updated the README

lanking520 · 2018-06-28T20:45:59Z

contrib/clojure-package/README.md

+### Deferred
+* Feed Forward API
+* OSX gpu support Scala - defer to adding via Scala first
+* CustomOp port - defer due to class loader issues


I am also working on improving this part, please let me know your issues, see if I can help

lanking520 · 2018-06-28T20:48:49Z

contrib/clojure-package/doc/getting-started/Ubuntu.md

+
+There are a few steps to get up and running with the dependencies on Ubuntu.
+
+Please see this issue for a handy guide: [Running on Ubuntu](https://github.com/gigasquid/clojure-mxnet/issues/2)


If any dependencies required for Scala, refer to this one. I think Aman is happy with the solution

lanking520 · 2018-06-28T21:06:26Z

contrib/clojure-package/examples/cnn-text-classification/README.md

+The dataset can be obtained here: [https://github.com/yoonkim/CNN_sentence](https://github.com/yoonkim/CNN_sentence). The two files `rt-polarity.neg`
+and `rt-polarity.pos` must be put in a directory. For example, `data/mr-data/rt-polarity.neg`.
+
+You also must download the glove word embeddings. The suggested one to use is the smaller 50 dimension one


Currently, I cannot reproduce the same accuracy that Python have. There may be changes on the Scala Training code.

- remove AWS reference - put in system specific build instructions with jars - clean up

gigasquid · 2018-06-28T22:44:02Z

Thanks @lanking520 for the review :) - In regards to the CNN text classification, the example at the default is running on a limited dataset (to fit in my laptop memory). There is an item on the Needs Contribution Help page about taking another look at that example with Word2Vec. I added to compare it to the python example results as well.

yzhliu

I'm more curious about the gen/ndarray(symbol).clj. @gigasquid Did you hand-write all these functions, or you have a way to auto-generate it?

yzhliu · 2018-06-29T06:40:11Z

contrib/clojure-package/README.md

+
+ Run `make scalapkg` then `make scalainstall`
+
+then replace the correct jar for your architecture in the project.clj, example `[ml.dmlc.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.3.0-SNAPSHOT"]`


org.apache instead?

Nice catch - will update

yzhliu · 2018-06-29T06:49:22Z

contrib/clojure-package/src/org/apache/clojure_mxnet/gen/ndarray.clj

+
+
+
+


in my understanding, you're listing all the operators below. Can Clojure auto-generate these functions? or can we have a function to allow users call by function name? e.g., NDArray.call("foo", arg1, arg2, ...,).

Currently, Clojure generated these by the write file method. I saw @gigasquid uploads the 3000+ lines of code on Symbol and NDArray in the Gen folder.

@yzhliu @lanking520 That is correct. We are autogenerating the code in the gen directories using the /dev/generator.clj code. This code gets invoking using a command line lein generate-code and produces these files by inspecting the classes created by Scala with java reflection. Where the scala package creates these classes using macros directly into memory without persisting, the clojure package chose to persist them to files to help the user discover what functions are available, generate api documentation, and to be able to see what new functions were added via a git diff from release to release.

Can we generate the source code file during compiling, instead of checking in to the repo?

Yes that is definitely a possibility to investigate. There are pros and cons to doing it manually vs compile time. For example, doing it as a manual step gives you a way to control the flow of changes into the project and a git diff. Where, doing it at compile time gives the advantage of not having to worry about a manual step. I'm definitely open to changing this aspect of it, especially if the manual generate-code step becomes a burden. I added it to the https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs as feature to further investigate. Thanks for the feedback :)

Currently on Scala side, @nswamy proposed a solution: Generate the signature of the file and documentation rather than the entire implementation. Maybe Clojure can follow the similar step to do. We can put this as a feature in the future.

That's a good idea. I added the detail of this to the Clojure Package Contribution Needs - thanks :)

@gigasquid I am also concerned that we are merging generated code, we do not want to merge massive amounts of generated code -- this is also a moving target as and when new operators are added or modified. The proposal to only have the interface/documentation also helps us to have a config driven implementation. Do you think you can prioritize this before 1.3?

@nswamy Of course. I can work on that next.

thanks so much :), you rock!

yzhliu · 2018-06-29T06:51:13Z

contrib/clojure-package/src/org/apache/clojure_mxnet/gen/symbol.clj

+
+
+
+


same question as that in ndarray.

It is the same process as described in the ndarray.

gigasquid · 2018-06-29T12:54:50Z

Thanks @yzhliu for your review. If you have any other questions, please let me know 😄

szha · 2018-06-30T20:10:20Z

@gigasquid and team, congrats! It's been a long way since @gigasquid pitched the idea of having a clojure frontend for mxnet since NIPS last year, and I'm thrilled to see this becoming the reality now. This effort opens up mxnet to the clojure enthusiasts, and I hope that we will attract more people from this community and see more interesting work coming soon :)

larroy · 2018-07-01T03:43:47Z

Let's merge this in unless somebody has additional comments.

nswamy · 2018-07-01T04:23:43Z

contrib/clojure-package/project.clj

+                 [t6/from-scala "0.3.0"]
+
+                 ;; Jars from Nexus
+                 ;[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.2.1"]


I am wondering if Clojure always depended on Maven jars, would the clojure-jars be always 1 release behind Scala? or are you consuming from the source build from the master?

The CI build is taking it from master when the jars are installed locally with make scalainstall. I kept the maven jars lines in there as well commented out so that it is easy for a user to switch back and forth.

@kurman

* Initial commit for clojure package - src code * examples for clojure package * Add cloverage code coverage plugin. Clean up dependencies in namespaces to make it compatible with the plugin * add bug fix from Kamil Hryniewicz * add basic optimizer test * add Kamil to thanks * add Christian Weilbach * add set -e to scripts * added documentation about the current process of building and deploying * unit tests for util * add tests for generator - Also fix parameter names to lowercase * make the test-validate use a spec in util-test * update to 1.2.1 scala version * add generator test fix & enhancements from Burin * add test for eval-metric * add callback unit test * switch project default to linux-cpu for CI - add thanks for Avram * remove reference to the clojars snapshot * @kurman feedback newline/scripts - add newline to gitignore - add -vx options to scripts * feedback from @kurman - use scala instead of $ * move gitignore back into clojure-package * feedback from @kurman - lowercase testing.md * feedback from @kurman use generate code alias for better clarity Supported APIs * feedback from @kurman - cnn text example * Add clojure package testing to CI * bug fix from jimdunn - skip validation in fit when eval-data is nil * Change CI to use locally installed scala jars instead of nexus * clean ci & bump clojure-package version to be inline with scala pacakge version -remove makefile modification and use script instead * update cat image link since the other one is broken * change cat link to one in the repo * Update README - remove AWS reference - put in system specific build instructions with jars - clean up * missing minus sign in example - contributed by jimdunn * feedback from @lanking520 * feedback from @lanking520 - specify that branch checkout is optional * feedback from @yzhliu - update README ml.dmlc -> org.apache

gigasquid mentioned this pull request Jun 8, 2018

Running on Ubuntu Linux gigasquid/clojure-mxnet#2

Open

szha requested a review from yzhliu June 8, 2018 20:18

gigasquid added 2 commits June 9, 2018 11:29

Initial commit for clojure package - src code

21b6d7b

examples for clojure package

d3961ff

gigasquid force-pushed the clojure-package branch from abb4a31 to d3961ff Compare June 9, 2018 15:30

Add cloverage code coverage plugin.

d4645b1

Clean up dependencies in namespaces to make it compatible with the plugin

gigasquid added 3 commits June 13, 2018 20:51

add bug fix from Kamil Hryniewicz

8d50031

add basic optimizer test

7b4b496

add Kamil to thanks

e7e1d27

gigasquid mentioned this pull request Jun 14, 2018

Ada delta fix gigasquid/clojure-mxnet#4

Merged

add Christian Weilbach

838330e

larroy reviewed Jun 16, 2018

View reviewed changes

gigasquid added 5 commits June 18, 2018 17:35

add set -e to scripts

8de88ed

added documentation about the current process of building and deploying

2f876ba

unit tests for util

948e24a

add tests for generator

d39cd66

- Also fix parameter names to lowercase

make the test-validate use a spec in util-test

ecd065b

larroy reviewed Jun 27, 2018

View reviewed changes

Change CI to use locally installed scala jars instead of nexus

edf795a

gigasquid requested a review from szha as a code owner June 27, 2018 02:14

gigasquid added 2 commits June 27, 2018 06:42

clean ci & bump clojure-package version to be inline with scala pacak…

29fbe28

…ge version -remove makefile modification and use script instead

update cat image link since the other one is broken

acfec15

gigasquid added 2 commits June 27, 2018 17:36

change cat link to one in the repo

69f9eb6

Merge branch 'master' into clojure-package

0863846

cjolivier01 reviewed Jun 28, 2018

View reviewed changes

marcoabreu approved these changes Jun 28, 2018

View reviewed changes

larroy reviewed Jun 28, 2018

View reviewed changes

larroy approved these changes Jun 28, 2018

View reviewed changes

lanking520 approved these changes Jun 28, 2018

View reviewed changes

gigasquid added 3 commits June 28, 2018 17:38

Update README

29485b8

- remove AWS reference - put in system specific build instructions with jars - clean up

missing minus sign in example - contributed by jimdunn

6567766

feedback from @lanking520

3669345

feedback from @lanking520 - specify that branch checkout is optional

fa9f23b

yzhliu reviewed Jun 29, 2018

View reviewed changes

feedback from @yzhliu - update README ml.dmlc -> org.apache

6760c90

szha approved these changes Jul 1, 2018

View reviewed changes

szha merged commit 8917318 into apache:master Jul 1, 2018

nswamy reviewed Jul 1, 2018

View reviewed changes

marcoabreu mentioned this pull request Jul 1, 2018

[MXNET-616] Add licenses to files introduced by clojure package #11205 #11512

Merged

6 tasks

danielcompton mentioned this pull request Jul 1, 2018

Clojure Library for mxnet #8971

Closed

gigasquid mentioned this pull request Jul 2, 2018

[MXNET-618] Clojure package - move NDArray and Symbol Code generation to Automatic #11534

Merged

5 tasks


		## Introduction

		MXNet is a first class, modern deep learning library that AWS has officially picked as its chosen library. It supports multiple languages on a first class basis and is incubating as an Apache project.


		Why build on the Scala package?

		The motivation section addresses this, but the main reason is high leverage is using the great work that the Scala package has already done.


		The motivation section addresses this, but the main reason is high leverage is using the great work that the Scala package has already done.

		How can I tell if the gpu is being used?


		There are a few steps to get up and running with the dependencies on Ubuntu.

		Please see this issue for a handy guide: [Running on Ubuntu](https://github.com/gigasquid/clojure-mxnet/issues/2)


		Run `make scalapkg` then `make scalainstall`

		then replace the correct jar for your architecture in the project.clj, example `[ml.dmlc.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.3.0-SNAPSHOT"]`

Clojure Contrib Package #11205

Clojure Contrib Package #11205

Conversation

gigasquid commented Jun 8, 2018 • edited Loading

Description

Checklist

Essentials

Changes

Comments

gigasquid commented Jun 8, 2018

lanking520 commented Jun 8, 2018 • edited Loading

kurman-zz commented Jun 8, 2018

gigasquid commented Jun 8, 2018

gigasquid commented Jun 9, 2018

kurman-zz commented Jun 12, 2018

kovasb commented Jun 14, 2018 • edited Loading

gigasquid commented Jun 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larroy left a comment

Choose a reason for hiding this comment

gigasquid commented Jun 16, 2018

larroy left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gigasquid Jun 27, 2018 • edited Loading

Choose a reason for hiding this comment

gigasquid commented Jun 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcoabreu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larroy left a comment

Choose a reason for hiding this comment

lanking520 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gigasquid commented Jun 28, 2018

yzhliu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gigasquid commented Jun 29, 2018

szha commented Jun 30, 2018

larroy commented Jul 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gigasquid commented Jun 8, 2018 •

edited

Loading

lanking520 commented Jun 8, 2018 •

edited

Loading

kovasb commented Jun 14, 2018 •

edited

Loading

gigasquid commented Jun 15, 2018 •

edited

Loading

larroy left a comment •

edited

Loading

gigasquid Jun 27, 2018 •

edited

Loading