Add docker_storage_to_overlay2 role #5216

mbarnes · 2017-08-25T15:45:25Z

This adds an adhoc playbook to change the Docker storage driver to overlay2.

Note, this makes no attempt to migrate existing containers/images. It momentarily stops docker.service and uses atomic storage reset to wipe all local Docker storage.

Tested on:

RHEL 7.4 (works)
CentOS 7.3 (always fails)
Fedora 26 (works)

Some necessary changes for OverlayFS to support SELinux security labels landed in RHEL 7.4, so I had the playbook do a version check for RedHat/CentOS distributions. Presumably this will work on CentOS 7.4 once it's released.

twiest · 2017-08-25T17:40:31Z

playbooks/adhoc/docker_storage_to_overlay2/docker_storage_to_overlay2.yml

+        state: started
+
+    - name: Check if storage driver is already overlay2
+      shell: docker info | grep --ignore-case 'storage driver:.*overlay2'


We try really hard to not use shell. It's a very, very powerful module, which security wise is a bit scary.

More information from the openshift-ansible best practices guide:
https://github.com/openshift/openshift-ansible/blob/master/docs/best_practices_guide.adoc#The-Ansible-command-module-SHOULD-be-used-instead-of-the-Ansible-shell-module

Could we instead use command?

Something like:

- name: Check if storage driver is already overlay2 command: docker info register: storage_driver_check changed_when: False ignore_errors: yes - name: Transition storage driver to overlay2 when: storage_driver_check.stdout | lower | search('storage driver:.*overlay2') block: [... snip ...]

Ah ha, thanks for the link. I'll rework to avoid shell.

twiest · 2017-08-25T17:42:44Z

playbooks/adhoc/docker_storage_to_overlay2/docker_storage_to_overlay2.yml

+          with_items: "{{ service_names }}"
+
+        - name: Check if storage driver is now overlay2
+          shell: docker info | grep --ignore-case 'storage driver:.*overlay2'


Same as above, consider switching to command instead of shell.

twiest · 2017-08-25T17:44:13Z

playbooks/adhoc/docker_storage_to_overlay2/docker_storage_to_overlay2.yml

+            state: absent
+          with_items:
+            - "EXTRA_STORAGE_OPTIONS"
+            - "EXTRA_DOCKER_STORAGE_OPTIONS"  # Deprecated


We use options like dm.basesize that I believe should work on both overlay and devicemapper.

Maybe allow us to pass these options in?

We need this option to be set (and possibly others).

dm.anything is devicemapper-specific, according to the docs.

The Docker daemon choked on it when I tried to pass it for overlay2:

$ systemctl status -l docker ... ... fatal_msg="Error starting daemon: error initializing graphdriver: overlay2: Unknown option dm.basesize\n" ... Failed to start Docker Application Container Engine. ...

The only documented option for overlay2 is overlay2.override_kernel_check, which we don't need.

Ah, interesting. So are we losing the ability to specify basesize?

If so, that's not good. What will keep a single container from eating up all of our docker storage volume?

@rhatdan: ^^ Can you answer this?

I'm iterating on your initial devicemapper-to-overlay2 playbook.

@twiest Did some Googling on your question and found this Moby PR.

The PR enables disk quota limits to be set per-container by way of a --storage-opt size option when using the overlay2 backend on an XFS filesystem:
e.g. docker run --storage-opt size=3g -it fedora /bin/bash

But the catch is the XFS filesystem must be mounted with project quota (pquota) support instead of the noquota default. And I don't see a way to inject that mount option by way of container-storage-setup.

Does this help at all, or is a limit-per-container a non-starter?

@mbarnes that is the functionality we want, but we'd need to ensure that whatever is starting the containers supports that. In this case, that'd mean openshift-node / CRI-O. As far as the pquota mount option, we'll need to have the container storage volume mounted via the fstab I assume, so we can work with that, I think.

@mrunalp Does CRI-O support setting --storage-opt size=3g on container start? We need a way to limit the amount of space a container can take on the container storage volume.

Submitted projectatomic/container-storage-setup#255 to see if we can get the pquota option into container-storage-setup.

The docker-storage-setup.service unit is what actually mounts the volume during boot, and I'd prefer to solve this in the tooling, but we might have to hack /etc/fstab as an interim solution.

Ah, ok, I didn't realize that.

I agree with you that it'd be better to fix in the tooling.

@twiest : I found what we need! Add overlay2.size daemon storage-opt

But this feature was only just added in June and isn't available where we need it to be. I emailed you and @rhatdan about it.

@mbarnes that's great, good find!

mwoodson · 2017-08-25T16:07:32Z

playbooks/adhoc/docker_storage_to_overlay2/docker_storage_to_overlay2.yml

+#
+
+- name: Transition Docker storage driver to overlay2
+  hosts: "tag_Name_{{ cli_tag_name }}"


My guess is this needs to use the openshift-ansible inventory format.

Yeah, I had no idea how to write that part. Was copying off some other adhoc playbook.

Can you elaborate what openshift-ansible inventory format looks like?

Yeah, this should either be hosts: nodes or we'd have to initialize the groups and then it'd be hosts: oo_nodes_to_config

Sorry, let me clarify, if we leave it in adhoc hosts: nodes is the right thing to do. If we expect this to be supported for OCP use we should initialize the groups in playbooks/byo/openshift-node/docker_storage_to_overlay2.yml and move the logic to playbooks/common/openshift-node/docker_storage_to_overlay2.yml

Okay, thanks for clarification, I'll make it hosts: nodes for now.

mbarnes · 2017-08-25T20:55:48Z

⬆️ Fixup commit addresses @twiest's comments.

I managed to avoid the shell module, but isolating text in a multiline string in Ansible is pretty tricky, it turns out. If you know of an easier way than what I'm doing here, do tell!

Other miscellaneous tweaks here too.

I'm still not sure what to do with the hosts: value...

mbarnes · 2017-08-28T15:38:14Z

I took a closer look at the final state of the system after running this playbook, and found that container-storage-setup was not mounting the device previously used for devicemapper despite specifying DEVS and VG in /etc/sysconfig/docker-storage-setup.

The contents of /var/lib/docker/overlay2 appear to be on the root logical volume.

Reading the https://github.com/projectatomic/container-storage-setup code, it looks like the setup tool is looking for a couple additional parameters for overlay2:

CONTAINER_ROOT_LV_NAME
CONTAINER_ROOT_LV_MOUNT_PATH

~~I'm currently trying to figure out how to set these properly.~~

Update: Looks like the best way to set these is via atomic storage modify with the --lvname and --rootfs options.

mbarnes · 2017-08-28T18:22:53Z

2nd fixup commit adds --lvname and --rootfs options to the atomic storage modify command. Now, the volume group previously used for devicemapper has a logical volume named docker-root-lv mounted at /var/lib/docker.

$ lsblk
NAME                           MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
...
vdb                            252:16   0   16G  0 disk 
└─vdb1                         252:17   0   16G  0 part 
  └─docker_vg-docker--root--lv 253:2    0  6.4G  0 lvm  /var/lib/docker

Does that look like the configuration we want for overlay2?

mbarnes · 2017-08-29T14:35:58Z

FTR, matched the generated configuration to what a subject-matter-expert recommends.
So I believe I'm on the right track.

mrunalp · 2017-08-31T14:54:20Z

@nalind PTAL

rhatdan · 2017-08-31T22:23:59Z

@mrunalp I think you meant @rhvgoyal PTAL

mbarnes · 2017-09-13T17:44:32Z

⬆️ Squashed the fixup commits and rebased on master with a few enhancements:

Document the overlay2 driver enhancements required for the overlay2.size option.
If present, capture the dm.basesize value and apply it to overlay2.size.

mbarnes · 2017-09-14T17:33:59Z

@mwoodson, @twiest: Also PTAL

mbarnes · 2017-09-20T15:44:17Z

⬆️ Added a test for container-storage-setup >= 0.7.0 which mounts with pquota enabled.

mbarnes · 2017-09-22T19:20:33Z

/test tox

mbarnes · 2017-09-25T16:40:09Z

⬆️ Squashed fixups and rebased again...

dak1n1 · 2017-10-02T14:24:29Z

playbooks/adhoc/docker_storage_to_overlay2/docker_storage_to_overlay2.yml

+#
+
+- name: Transition Docker storage driver to overlay2
+  hosts: nodes


Just checking -- this will also update the driver on the masters, right? Ops would like the storage driver configurations to be consistent across masters and nodes.

@dak1n1 Ah okay, wasn't clear to me what to use for hosts.

Should this then be:

hosts: - masters - nodes

?

dak1n1 · 2017-10-02T15:33:06Z

playbooks/adhoc/docker_storage_to_overlay2/docker_storage_to_overlay2.yml

+    # XXX Is there no easier way to do this?
+    - name: Get {{ item.key }} version
+      command: rpm --query --queryformat="%{version}-%{release}" "{{ item.key }}"
+      args:


We have a module we prefer to use for this, repoquery. For example:

- name: Get {{ item.key }} version repoquery: name: "{{ item.key }}" query_type: installed show_duplicates: True register: package_version

And I believe the data you want would be in package_version.results.versions.available_versions_full.

(This is more of a "nice to have" feature, btw. It doesn't have to hold up the stg testing.)

dak1n1 · 2017-10-02T15:40:36Z

playbooks/adhoc/docker_storage_to_overlay2/docker_storage_to_overlay2.yml

+# Note, this makes no attempt to migrate existing containers/images.
+# It momentarily stops docker.service and uses "atomic storage reset"
+# to wipe all local Docker storage.
+#


I just noticed this comment. How about if this playbook was a role instead, so Ops could call it after we migrate containers onto new nodes? Then we could control how many hosts have their storage wiped at once, and make sure users' apps remain running on other nodes. (Otherwise this would be too much of an outage to do in production). @mwoodson @jupierce thoughts?

@sdodson Do you guys have an existing framework we could plug destructive roles like this into? For example, it would drain 10% of nodes at a time, and then run whatever role you need. Like a yum update+reboot role, or this docker storage reset role.

(I have something that I use for that, but I'm supposed to check upstream before making custom playbooks).

@sdodson unping. Turns out we have approval to just use my playbook as a wrapper for this.

@dak1n1 Your playbook just wraps the group evaluation or have you moved this into a role?

dak1n1 · 2017-10-02T15:48:27Z

playbooks/adhoc/docker_storage_to_overlay2/docker_storage_to_overlay2.yml

+      command: atomic storage reset
+
+    - name: Capture storage setup configuration
+      command: cat "{{ storage_setup_path }}"


Another "nice to have"... Ops likes everything to be in module form if possible, rather than commands. The slurp module will do what you need for this task. https://docs.ansible.com/ansible/latest/slurp_module.html

mbarnes · 2017-10-03T00:14:23Z

@dak1n1 ⬆️ Addressed all the "nice to haves". Awaiting guidance on whether/how to convert this to a role, like you mentioned above.

sdodson · 2017-10-03T20:22:43Z

playbooks/adhoc/docker_storage_to_overlay2/docker_storage_to_overlay2.yml

+        that: ansible_distribution_version | version_compare("7.4", "ge")
+        msg: "The OverlayFS file system only works with SELinux in {{ ansible_distribution }} 7.4 or later."
+
+    - name: Get "{{ item.key }}" version


Wrap these two tasks in a block that only fires when a quota is specified so that they can migrate to overlayfs without quotas if they don't meet the version requirements?

- block: when: disk_quota|length > 0

mbarnes · 2017-10-04T14:32:04Z

⬆️ Implemented @sdodson's suggestion.

Working now on converting this all to a role so it can be wrappered for ops.

mbarnes · 2017-10-04T17:52:36Z

⬆️ Rebased again and converted the whole thing to a role.

@sdodson, @dak1n1: 1st time writing an Ansible role, so further review is much appreciated.

kwoodson · 2017-11-02T19:18:53Z

@ashcrow, @michaelgugino,

Please weigh in on this. This is the migration from device-mapper to overlay for docker.

ashcrow

It's looking good. Added a few notes.

ashcrow · 2017-11-02T19:42:23Z

roles/docker_storage_to_overlay2/defaults/main.yml

@@ -0,0 +1,3 @@
+---
+container_storage_lvname: "docker-root-lv"
+container_storage_lvsize: "100%FREE"


Googling this gives some weird results!

ashcrow · 2017-11-02T19:43:35Z

roles/docker_storage_to_overlay2/tasks/main.yml

+
+- name: Capture storage setup configuration
+  slurp:
+    src: "{{ storage_setup_path }}"


@kwoodson as in:

role_storage_setup_path

or

docker_storage_to_overlay2_storage_setup_path

ashcrow · 2017-11-02T19:45:59Z

roles/docker_storage_to_overlay2/vars/main.yml

+service_names: []
+
+optional_service_names:
+- "docker.service"            # } One or the other


No change needed but openshift_facts does set openshift.docker.service_name which will expand to either docker.service or container-enginer.service.

👍 Nice, that's handy. Also see my question above about docker info.

ashcrow · 2017-11-02T19:46:50Z

roles/docker_storage_driver/tasks/main.yml

+
+- name: Start docker.service
+  service:
+    name: docker


openshift.docker.service_name might be used here if container-engine is also supported.

Yeah, about that. What would be the equivalent to docker info when using the container-engine system container?

In my playbook that uses this role, I run docker info at the beginning and extract the "Storage Driver" value. If it's already "overlay2" I skip the whole thing so the playbook is idempotent. I'd like to preserve that feature when using container-engine.

docker info should be the same. docker client is installed and will use container-engine service.

ashcrow · 2017-11-02T20:17:06Z

I really want to see this make it into the installer. This would make use of overlay, which is becoming common, much easier.

ashcrow · 2017-11-03T18:39:45Z

roles/docker_storage_driver/tasks/main.yml

  service:
-    name: docker
+    name: "{{ openshift.docker.service_name }}"


ashcrow · 2017-11-03T18:42:23Z

roles/docker_storage_to_overlay2/vars/main.yml

-optional_service_names:
- "docker.service"            # } One or the other
- "container-engine.service"  # } must be present
+docker_storage_to_overlay2_optional_service_names:


Note that when origin is installed the service is called origin-node.service You can use {{ openshift.common.service_type }} to get the proper one.

mbarnes · 2017-11-03T19:11:36Z

General question: Does Ansible (or openshift-ansible) provide a module for rebooting a host and waiting for its SSH service to come back up? I copied someone's Stack Overflow solution for this, but I'm surprised it's not built-in. Seems basic for an automation system.

sdodson · 2017-11-03T19:23:58Z

@mbarnes here's the task we use for this else where, the waiting for the api you don't need.
https://github.com/openshift/openshift-ansible/blob/master/playbooks/common/openshift-master/restart_hosts.yml

ashcrow · 2017-11-03T19:24:50Z

@mbarnes Check out playbooks/common/openshift-master/restart_hosts.yml which is likely similar to what you found online.

mbarnes · 2017-11-03T19:35:03Z

@sdodson, @ashcrow: Thanks, that at least looks similar to what I'm doing. I need to reboot non-master nodes too, so I guess I'll stick with what I have for now.

ashcrow · 2017-11-06T19:13:19Z

@giuseppe PTAL as well.

michaelgugino

I think we should support overlay2, but I don't think we should support it in an ad-hoc manner. I think this should be a supported storage option in-line with how we support other storage options.

We should not do any hard-removing of existing storage. This should be an up-front decision, and if a node needs to go from one docker storage type to another, that node needs to be reconfigured from scratch.

Let's hold off on this, and pick back up when we start work on 3.8.

michaelgugino · 2017-11-06T19:24:21Z

roles/docker_storage_driver/meta/main.yml

@@ -0,0 +1,3 @@
+---
+dependencies:
+- openshift_facts


I'm hoping to refactor all of the docker portions out of openshift_facts. That almost shipped this cycle, but slipped due to feature freeze.

michaelgugino · 2017-11-06T19:25:11Z

roles/docker_storage_driver/tasks/main.yml

@@ -0,0 +1,19 @@
+---


We're stripping out the docker portions of openshift_facts.

I agree, this doesn't need to be it's own role.

mbarnes · 2017-11-06T20:02:35Z

We should not do any hard-removing of existing storage. This should be an up-front decision, and if a node needs to go from one docker storage type to another, that node needs to be reconfigured from scratch.

Just to add context here, I'm writing this for the OpenShift Online operations team as a prerequisite for moving to CRI-O, and for the Online clusters I wrappered this role with a playbook that drains each node before doing the storage conversion.

I was planning on trying to adapt that playbook for inclusion here, unless you think it's the wrong approach.

michaelgugino · 2017-11-06T20:01:06Z

roles/docker_storage_driver/tasks/main.yml

@@ -0,0 +1,19 @@
+---


Yes, my intention is to remove everything openshift.docker and remove all openshift.x variables from the docker roles.

michaelgugino · 2017-11-06T20:02:43Z

roles/docker_storage_to_overlay2/vars/main.yml

@@ -0,0 +1,32 @@
+---


My preference is to put things in defaults/main.yml unless there's a specific reason not to.

michaelgugino · 2017-11-06T20:10:43Z

I was planning on trying to adapt that playbook for inclusion here, unless you think it's the wrong approach.

I think it's best to leave that kind of playbook in contrib or a related repo. I don't think we should have playbooks that wipe away containers and storage as it just opens the door for users to misapply those playbooks. This would put a large burden of support on these plays to build tons of safeguards and any regressions could have catastrophic effects.

I would assume that these nodes need to be drained, configured, and re-added, so that seems like someone might as well rebuild the node or add new nodes and retire the old ones.

ashcrow · 2017-11-10T16:42:52Z

@mbarnes let @michaelgugino know when you are ready to have this re-reviewed.

mbarnes · 2017-11-13T12:49:00Z

@mbarnes let @michaelgugino know when you are ready to have this re-reviewed.

I think I fixed or commented on everything mentioned, so it's ready for re-review.

I still think it makes sense to merge the docker_storage_driver role into some Docker fact collecting role -- whether that's openshift_facts or a separate docker_facts or whatever else @michaelgugino had in mind -- since I found a 2nd use case for it (grow_docker_vg).

I wasn't privy to all the discussions about this last week. Is current thinking that just the role is being considered, or should I resume trying to add some node-draining wrapper playbook like what I'm currently using for Openshift Online?

ashcrow · 2017-11-13T15:44:59Z

@michaelgugino ⬆️

sdodson · 2017-11-13T15:52:15Z

https://trello.com/c/IGCV4VRH/561-add-devicemapper-to-overlayfs-migration-playbook card for CL team to track getting this into 3.9 in a supported manner.

michaelgugino · 2017-11-13T18:23:55Z

I added this topic as a discussion item for this week's architecture meeting.

ashcrow · 2017-11-13T18:30:06Z

Don't forget to explicitly invite @mbarnes to the meeting!

ashcrow · 2017-12-01T19:07:08Z

What's the status of this PR post meeting?

sdodson · 2017-12-01T20:36:37Z

What's the status of this PR post meeting?

reviewing my notes we said we'd put in a card to deliver this for 3.9 in a supported manner.

https://trello.com/c/IGCV4VRH/561-3-add-devicemapper-to-overlayfs-migration-playbook

ashcrow · 2017-12-01T21:03:01Z

@sdodson ok thanks!

Returns the current Docker storage driver according to "docker info". Sets fact: "docker_storage_driver"

This role changes the Docker storage driver to overlay2. Note, this makes no attempt to migrate existing containers/images. It momentarily stops docker.service and uses "atomic storage reset" to wipe all local Docker storage.

openshift-ci-robot · 2018-12-13T18:32:35Z

@mbarnes: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/logging	`1cb7c07`	link	`/test logging`
ci/prow/e2e-aws	`1cb7c07`	link	`/test e2e-aws`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

kwoodson · 2018-12-13T18:43:52Z

I'd recommend closing this as its been over a year.

vrutkovs · 2018-12-13T18:47:13Z

Master branch is closed! A major refactor is ongoing in devel-40. Changes for 3.x should be made directly to the latest release branch they're relevant to and backported from there.

mbarnes changed the title ~~playbooks/adhoc: Add docker_storage_to_overlay2.yml~~ [WIP] playbooks/adhoc: Add docker_storage_to_overlay2.yml Aug 25, 2017

twiest suggested changes Aug 25, 2017

View reviewed changes

mwoodson reviewed Aug 25, 2017

View reviewed changes

mbarnes mentioned this pull request Aug 30, 2017

"atomic storage reset" fails if /var/lib/docker is mounted projectatomic/container-storage-setup#254

Open

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 13, 2017

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 25, 2017

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 25, 2017

dak1n1 reviewed Oct 2, 2017

View reviewed changes

sdodson reviewed Oct 3, 2017

View reviewed changes

kwoodson requested a review from michaelgugino November 2, 2017 19:18

ashcrow reviewed Nov 2, 2017

View reviewed changes

ashcrow reviewed Nov 3, 2017

View reviewed changes

michaelgugino suggested changes Nov 6, 2017

View reviewed changes

mbarnes added 2 commits December 13, 2017 13:33

Add docker_storage_driver role

a2935e6

Returns the current Docker storage driver according to "docker info". Sets fact: "docker_storage_driver"

Add docker_storage_to_overlay2 role

1cb7c07

This role changes the Docker storage driver to overlay2. Note, this makes no attempt to migrate existing containers/images. It momentarily stops docker.service and uses "atomic storage reset" to wipe all local Docker storage.

ashcrow mentioned this pull request Dec 19, 2017

Adding support for docker-storage-setup on overlay #6524

Merged

vrutkovs closed this Dec 13, 2018

Add docker_storage_to_overlay2 role #5216

Add docker_storage_to_overlay2 role #5216

Conversation

mbarnes commented Aug 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbarnes Aug 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbarnes commented Aug 25, 2017

mbarnes commented Aug 28, 2017 • edited Loading

mbarnes commented Aug 28, 2017 • edited Loading

mbarnes commented Aug 29, 2017 • edited Loading

mrunalp commented Aug 31, 2017

rhatdan commented Aug 31, 2017

mbarnes commented Sep 13, 2017 • edited Loading

mbarnes commented Sep 14, 2017

mbarnes commented Sep 20, 2017

mbarnes commented Sep 22, 2017

mbarnes commented Sep 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dak1n1 Oct 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbarnes commented Oct 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbarnes commented Oct 4, 2017

mbarnes commented Oct 4, 2017

kwoodson commented Nov 2, 2017

ashcrow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbarnes Nov 2, 2017 • edited Loading

Choose a reason for hiding this comment

ashcrow Nov 2, 2017 • edited Loading

Choose a reason for hiding this comment

ashcrow commented Nov 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbarnes commented Nov 3, 2017

sdodson commented Nov 3, 2017

ashcrow commented Nov 3, 2017

mbarnes commented Nov 3, 2017

ashcrow commented Nov 6, 2017

michaelgugino left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbarnes commented Nov 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelgugino commented Nov 6, 2017

ashcrow commented Nov 10, 2017

mbarnes commented Nov 13, 2017 • edited Loading

ashcrow commented Nov 13, 2017

mbarnes Aug 29, 2017 •

edited

Loading

mbarnes commented Aug 28, 2017 •

edited

Loading

mbarnes commented Aug 28, 2017 •

edited

Loading

mbarnes commented Aug 29, 2017 •

edited

Loading

mbarnes commented Sep 13, 2017 •

edited

Loading

dak1n1 Oct 2, 2017 •

edited

Loading

mbarnes Nov 2, 2017 •

edited

Loading

ashcrow Nov 2, 2017 •

edited

Loading

mbarnes commented Nov 6, 2017 •

edited

Loading

mbarnes commented Nov 13, 2017 •

edited

Loading