Unable to provision using archivematica playbook with vars-singlenode-qa.yml (on Windows) #40

minusdavid · 2017-08-07T02:34:18Z

I've tried to provision a few times now using vars-singlenode-qa.yml instead of vars-singlenode-1.6.yml, and I keep getting the same error.

TASK [artefactual.archivematica-src : Checkout archivematica-storage-service code] ***
fatal: [am-local]: FAILED! => {"changed": false, "cmd": "/usr/bin/git checkout --track -b qa/0.x origin/qa/0.x", "failed": true, "msg": "Failed to checkout qa/0.x", "rc": 1, "stderr": "error: Your local changes to the following files would be overwritten by checkout:\n\t
storage_service/static/js/vendor/base64.js\nPlease, commit your changes or stash them before you can switch branches.\nAborting\n", "stderr_lines": ["error: Your local changes to the following files would be overwritten by checkout:", "\tstorage_service/static/js/vendor/
base64.js", "Please, commit your changes or stash them before you can switch branches.", "Aborting"], "stdout": "", "stdout_lines": []}
to retry, use: --limit @/vagrant/singlenode.retry

When I look at src/archivematica-storage-service, I see that branch stable/0.10.x is checked out even though archivematica_src_ss_version is set to "qa/0.x".

roles/artefactual.archivematica-src/tasks/ss-main.yml looks good. The version corresponds to the yml file and force is set to "yes".

Investigating...

minusdavid · 2017-08-07T02:42:27Z

Here's the output from git after I try /usr/bin/git checkout --track -b qa/0.x origin/qa/0.x by hand:

vagrant@am-local:/vagrant/src/archivematica-storage-service$ git status
On branch stable/0.10.x
Your branch is up-to-date with 'origin/stable/0.10.x'.

Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

    modified:   osdeps/RedHat-7.json
    modified:   osdeps/Ubuntu-16.json
    modified:   storage_service/static/js/vendor/base64.js

no changes added to commit (use "git add" and/or "git commit -a")
vagrant@am-local:/vagrant/src/archivematica-storage-service$ git diff
diff --git a/osdeps/RedHat-7.json b/osdeps/RedHat-7.json
--- a/osdeps/RedHat-7.json
+++ b/osdeps/RedHat-7.json
@@ -1 +0,0 @@
-CentOS-7.json
\ No newline at end of file
diff --git a/osdeps/Ubuntu-16.json b/osdeps/Ubuntu-16.json
--- a/osdeps/Ubuntu-16.json
+++ b/osdeps/Ubuntu-16.json
@@ -1 +0,0 @@
-Ubuntu-14.json
\ No newline at end of file
diff --git a/storage_service/static/js/vendor/base64.js b/storage_service/static/js/vendor/base64.js
--- a/storage_service/static/js/vendor/base64.js
+++ b/storage_service/static/js/vendor/base64.js
@@ -1 +0,0 @@
-../../../external/base64-helpers/base64-helpers.js
\ No newline at end of file

--

vagrant@am-local:/vagrant/src/archivematica-storage-service$ ls -l storage_service/static/js/vendor/base64.js
lrwxrwxrwx 1 vagrant vagrant 0 Aug 7 02:16 storage_service/static/js/vendor/base64.js -> ../../../external/base64-helpers/base64-helpers.js

--

vagrant@am-local:/vagrant/src/archivematica-storage-service$ ls ../../../external/base64-helpers/base64-helpers.js
ls: cannot access ../../../external/base64-helpers/base64-helpers.js: No such file or directory

--

vagrant@am-local:/vagrant/src/archivematica-storage-service$ ls -l osdeps
total 3
-rwxrwxrwx 1 vagrant vagrant 1032 Aug 7 02:16 CentOS-7.json
lrwxrwxrwx 1 vagrant vagrant 0 Aug 7 02:16 RedHat-7.json -> CentOS-7.json
-rwxrwxrwx 1 vagrant vagrant 895 Aug 7 02:16 Ubuntu-14.json
lrwxrwxrwx 1 vagrant vagrant 0 Aug 7 02:16 Ubuntu-16.json -> Ubuntu-14.json

--

I'm guessing that maybe this is the Windows symlink issue raising its head again... even though in the case of "osdeps", the symlinks have been created successfully...

minusdavid · 2017-08-07T02:48:53Z

If I look at it using Git Bash, everything looks OK. It's only when I go into the VM that I see those differences and even using "git checkout -- " won't get rid of them.

minusdavid · 2017-08-07T02:53:07Z

Looks like the ../../../external/base64-helpers/base64-helpers.js might be submodule?

I'm going to try stable again.....

minusdavid · 2017-08-07T03:20:03Z

Using vars-singlenode-1.6.yml, and I'm still in the process of provisioning, but the archivematica-storage-service seems to have been installed correctly.

Here's the view from Windows:

$ ls -l
total 8
-rw-r--r-- 1 dcook 197121 1032 Aug 7 13:07 CentOS-7.json
lrwxrwxrwx 1 dcook 197121 13 Aug 7 13:07 RedHat-7.json -> CentOS-7.json
-rw-r--r-- 1 dcook 197121 895 Aug 7 13:07 Ubuntu-14.json
lrwxrwxrwx 1 dcook 197121 14 Aug 7 13:07 Ubuntu-16.json -> Ubuntu-14.json

$ ls -l storage_service/static/js/vendor/base64.js
lrwxrwxrwx 1 dcook 197121 50 Aug 7 13:07 storage_service/static/js/vendor/base64.js -> ../../../external/base64-helpers/base64-helpers.js

Going to see if I can vagrant ssh in during provisioning...

vagrant@am-local:/vagrant/src/archivematica-storage-service$ git status
On branch stable/0.10.x
Your branch is up-to-date with 'origin/stable/0.10.x'.

Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

    modified:   osdeps/RedHat-7.json
    modified:   osdeps/Ubuntu-16.json
    modified:   storage_service/static/js/vendor/base64.js

I'm seeing the same thing as last time... however git didn't fail when using vars-singlenode-1.6.yml for the archivematica-storage-service...

minusdavid · 2017-08-07T03:44:23Z

Hmm I notice that core.symlinks isn't set on the VM-side whereas I do have it set on the Windows side. I might experiment with that for a moment...

minusdavid · 2017-08-07T03:45:16Z

Hmm doesn't seem to make a difference at a glance.

minusdavid · 2017-08-07T03:56:49Z

I imagine this doesn't affect vars-singlenode-1.6.yml because git doesn't need to switch branches since the git repositories use the stable branches as the repository HEAD.

minusdavid · 2017-08-07T04:25:26Z

Because everything looks OK on Windows, I'm wondering if this is actually a vboxfs issue?

minusdavid · 2017-08-08T03:57:59Z

I noticed the Ansible command was "/usr/bin/git checkout --track -b qa/0.x origin/qa/0.x" which doesn't mention force... so I just tried the following in "archivematica":

"git checkout --force qa/1.x"

And that worked. The "git status" output has changed from:

minusdavid · 2017-08-08T03:58:14Z

    modified:   src/MCPClient/osdeps/RedHat-7.json
    modified:   src/MCPServer/osdeps/RedHat-7.json
    modified:   src/MCPServer/osdeps/Ubuntu-16.json
    modified:   src/archivematicaCommon/osdeps/RedHat-7.json
    modified:   src/archivematicaCommon/osdeps/Ubuntu-16.json
    modified:   src/dashboard/osdeps/RedHat-7.json
    modified:   src/dashboard/osdeps/Ubuntu-16.json
    modified:   src/dashboard/src/fpr
    modified:   src/dashboard/src/templates/fpr

to:

    modified:   src/MCPClient/osdeps/RedHat-7.json
    modified:   src/MCPServer/osdeps/RedHat-7.json
    modified:   src/MCPServer/osdeps/Ubuntu-16.json
    modified:   src/archivematicaCommon/osdeps/RedHat-7.json
    modified:   src/archivematicaCommon/osdeps/Ubuntu-16.json
    modified:   src/dashboard/osdeps/RedHat-7.json
    modified:   src/dashboard/osdeps/Ubuntu-16.json

minusdavid · 2017-08-08T04:00:04Z

@sevein mentioned that qa/1.x moved fpr out of a submodule and instead gets fpr using pip so that makes sense that it's gone. I imagine the others stayed the same across versions.

And they're only showing up as modified because vboxsf or at least vboxsf on Windows is maybe cloning them as a file with the symlink target as its content before creating the symlinks? I don't know enough about Git internals to know exactly what's going on here...

minusdavid · 2017-08-08T04:12:53Z

So I just tried the following:

cd /vagrant
git clone https://github.com/artefactual/archivematica.git test
touch /home/vagrant/test
ln -s ~/test /vagrant/test/test
git add test

At the end of a git diff, I see this:

diff --git a/test b/test
--- a/test
+++ b/test
@@ -1 +0,0 @@
-/home/vagrant/test
\ No newline at end of file

minusdavid · 2017-08-08T04:25:19Z

Looks like someone else noted this years ago:

https://stackoverflow.com/questions/34225522/files-changed-directly-after-git-clone-into-shared-folder-using-vagrant-on-windo

I'm thinking it must be something like... it gets cloned into the shared folder as a file containing text to the target, and then underneath Virtualbox Windows is creating a symlink after Git has already recorded it as a regular file?

Of course, it's not an issue on Windows. Git seems to work correctly there.

On Windows, I'm using git version 2.13.2.windows.1. Whereas in Virtualbox, I'm using git version 1.9.1.

Using the following to install a newer version of git:
sudo apt-add-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git

Now I'm using git version 2.13.0 on Virtualbox guest...

Still a problem.

--

Using "git update-index --assume-unchanged" might be an option, but not really in this case...
https://stackoverflow.com/questions/18771926/persuading-git-to-handle-symlinks-correctly-on-vboxsf-filesystem

minusdavid · 2017-08-08T04:26:30Z

"Support for symbolic links across synced folder implementations and host/guest combinations is not consistent. Vagrant does its best to make sure symbolic links work by configuring various hypervisors (such as VirtualBox), but some host/guest combinations still do not work properly. This can affect some development environments that rely on symbolic links.

The recommendation is to make sure to test symbolic links on all the host/guest combinations you sync folders on if this is important to you."

https://www.vagrantup.com/docs/synced-folders/basic_usage.html

But the symlinks do work... it's just git which is having an issue with it...

minusdavid · 2017-08-08T04:36:35Z

Tried the "git update-index --assume-unchanged" with mixed results. It mostly just acts to confuse git (and you).

minusdavid · 2017-08-08T04:43:28Z

So this is a problem... and it looks like it's probably due to having Windows as the host and Linux as the guest. While the symlinks do work, there's some issue with how they're created and how Git records their creation I guess. I'm not sure.

As for the actual problem I'm having... I think it's just that Ansible isn't having Git force the checkout despite the use of "force: yes" in the Ansible playbook.

This seems to be reported by others as well: ansible/ansible#19809

minusdavid · 2017-08-08T04:59:28Z

Looks like a bug in the "git" module in Ansible. I've highlighted it on that Ansible issue, but basically it looks like it ignores the "force" parameter because it's trying to check out a remote branch instead of a local branch.

minusdavid · 2017-08-08T05:12:26Z

Because I'm stubborn, I'm going to try to work around this...

I've gone into src/archivematica and src/archivematica-storage-service and manually created local branches for qa/1.x and qa/0.x respectively.

I'm guessing that will work... because that should avoid the "force" bug in Ansible.

minusdavid · 2017-08-08T05:23:01Z

Waiting for Ansible to run...

I see that other people have had issues with symlinks in other contexts: https://discuss.atom.io/t/atom-on-windows-incorrectly-showing-files-as-modified-git/12805/13

Hmm... the issue at https://www.bountysource.com/issues/39638459-lfs-tracked-files-are-sometimes-incorrectly-shown-as-modified-in-git-status-output has me thinking... if the issue is with the index... maybe there's something we can do. Still reading through this link though.

minusdavid · 2017-08-08T05:30:57Z

Note that my workaround of manually checking out qa/1.x and qa/0.x into src/archivematica and src/archivematica-storage-service worked...

However I encountered a different error during "vagrant provision":

TASK [artefactual.archivematica-src : Run SS django collectstatic] *************
fatal: [am-local]: FAILED! => {"changed": false, "cmd": "./manage.py collectstatic --noinput", "failed": true, "msg": "\n:stderr: Traceback (most recent call last):\n File "./manage.py", line 10, in \n execute_from_command_
line(sys.argv)\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/django/core/management/init.py", line 354, in execute_from_command_line\n utility.execute()\n File "/usr/share/python/ar
chivematica-storage-service/local/lib/python2.7/site-packages/django/core/management/init.py", line 346, in execute\n self.fetch_command(subcommand).run_from_argv(self.argv)\n File "/usr/share/python/archivematica-storage-servi
ce/local/lib/python2.7/site-packages/django/core/management/init.py", line 182, in fetch_command\n settings.INSTALLED_APPS\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/django/conf/__
init__.py", line 48, in getattr\n self._setup(name)\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/django/conf/init.py", line 44, in _setup\n self.wrapped = Settings(settings
module)\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/django/conf/init.py", line 92, in init\n mod = importlib.import_module(self.SETTINGS_MODULE)\n File "/usr/lib/python2.7/imp
ortlib/init.py", line 37, in import_module\n import(name)\n File "/vagrant/src/archivematica-storage-service/storage_service/storage_service/settings/production.py", line 33, in \n ALLOWED_HOSTS = get_env_variab
le('DJANGO_ALLOWED_HOSTS').split(',')\n File "/vagrant/src/archivematica-storage-service/storage_service/storage_service/settings/base.py", line 19, in get_env_variable\n raise ImproperlyConfigured(error_msg)\ndjango.core.exception
s.ImproperlyConfigured: Set the DJANGO_ALLOWED_HOSTS environment variable\n", "path": "/usr/share/python/archivematica-storage-service/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "state": "absent", "syspa
th": ["/tmp/ansible_kZZCwX", "/tmp/ansible_kZZCwX/ansible_modlib.zip", "/tmp/ansible_kZZCwX/ansible_modlib.zip", "/usr/lib/python2.7", "/usr/lib/python2.7/plat-x86_64-linux-gnu", "/usr/lib/python2.7/lib-tk", "/usr/lib/python2.7/lib-old",
"/usr/lib/python2.7/lib-dynload", "/usr/local/lib/python2.7/dist-packages", "/usr/lib/python2.7/dist-packages"]}
to retry, use: --limit @/vagrant/singlenode.retry

When I prettify that a bit, it looks like it's a configuration issue? So I'm guessing a problem with the qa/0.x branch?

minusdavid · 2017-08-08T05:31:36Z

If someone could on a non-Windows system could run deploy-pub using vars-singlenode-qa.yml, we might see if it is a more widespread problem there.

minusdavid · 2017-08-08T05:44:59Z

As for the git issue...

vagrant@am-local:/vagrant/src/archivematica-storage-service$ git ls-tree HEAD osdeps/Ubuntu-16.json
120000 blob ab357c396a99f5381083bd12666da5d464939a32 osdeps/Ubuntu-16.json

vagrant@am-local:/vagrant/src/archivematica-storage-service$ git cat-file -p ab357c396a99f5381083bd12666da5d464939a32
Ubuntu-14.json

vagrant@am-local:/vagrant/src/archivematica-storage-service$ git show HEAD:osdeps/Ubuntu-16.json
Ubuntu-14.json

vagrant@am-local:/vagrant/src/archivematica-storage-service$ cat osdeps/Ubuntu-16.json
<actual contents of the osdeps/Ubuntu-14.json>

vagrant@am-local:/vagrant/src/archivematica-storage-service$ git diff osdeps/Ubuntu-16.json
diff --git a/osdeps/Ubuntu-16.json b/osdeps/Ubuntu-16.json
--- a/osdeps/Ubuntu-16.json
+++ b/osdeps/Ubuntu-16.json
@@ -1 +0,0 @@
-Ubuntu-14.json
\ No newline at end of file

--

Interestingly, on Linux, I see some of the same stuff in git:
dcook@linux:~/git/archivematica/src/MCPServer/osdeps> git show HEAD:src/MCPServer/osdeps/Ubuntu-16.json
Ubuntu-14.json

But it doesn't show up as modified in "git status".

But yeah... that's a level of git beyond my current understanding.

minusdavid · 2017-08-08T06:14:09Z

I've removed archivematica_src_dir from the Vagrantfile so I'm using /opt/archivematica instead of /vagrant/src to avoid the issues with Ansible's buggy git module and Git's buggy handling of symlinks for a Windows host/Linux guest Virtualbox shared folder.

However, I'm still running into this problem, so I'm thinking that there's an issue with archivematica-storage-service's qa/0.x branch.

TASK [artefactual.archivematica-src : Run SS django collectstatic] *************
fatal: [am-local]: FAILED! => {"changed": false, "cmd": "./manage.py collectstatic --noinput", "failed": true, "msg": "\n:stderr: Traceback (most recent call last):\n File "./manage.py", line 10, in \n execute_from_command_
line(sys.argv)\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/django/core/management/init.py", line 354, in execute_from_command_line\n utility.execute()\n File "/usr/share/python/ar
chivematica-storage-service/local/lib/python2.7/site-packages/django/core/management/init.py", line 346, in execute\n self.fetch_command(subcommand).run_from_argv(self.argv)\n File "/usr/share/python/archivematica-storage-servi
ce/local/lib/python2.7/site-packages/django/core/management/init.py", line 182, in fetch_command\n settings.INSTALLED_APPS\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/django/conf/__
init__.py", line 48, in getattr\n self._setup(name)\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/django/conf/init.py", line 44, in setup\n self.wrapped = Settings(settings
module)\n File "/usr/share/python/archivematica-storage-service/local/lib/python2.7/site-packages/django/conf/init.py", line 92, in init\n mod = importlib.import_module(self.SETTINGS_MODULE)\n File "/usr/lib/python2.7/imp
ortlib/init.py", line 37, in import_module\n import(name)\n File "/opt/archivematica/archivematica-storage-service/storage_service/storage_service/settings/production.py", line 33, in \n ALLOWED_HOSTS = get_env
variable('DJANGO_ALLOWED_HOSTS').split(',')\n File "/opt/archivematica/archivematica-storage-service/storage_service/storage_service/settings/base.py", line 19, in get_env_variable\n raise ImproperlyConfigured(error_msg)\ndjango.co
re.exceptions.ImproperlyConfigured: Set the DJANGO_ALLOWED_HOSTS environment variable\n", "path": "/usr/share/python/archivematica-storage-service/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "state": "abs
ent", "syspath": ["/tmp/ansible_HV1PHp", "/tmp/ansible_HV1PHp/ansible_modlib.zip", "/tmp/ansible_HV1PHp/ansible_modlib.zip", "/usr/lib/python2.7", "/usr/lib/python2.7/plat-x86_64-linux-gnu", "/usr/lib/python2.7/lib-tk", "/usr/lib/python2
.7/lib-old", "/usr/lib/python2.7/lib-dynload", "/usr/local/lib/python2.7/dist-packages", "/usr/lib/python2.7/dist-packages"]}
to retry, use: --limit @/vagrant/singlenode.retry

jraddaoui · 2017-08-08T15:45:45Z

I'm getting the same error in Linux. It looks like a new env var is required (in production?) after artefactual/archivematica-storage-service@8934b22

jraddaoui · 2017-08-08T16:42:10Z

The following PR should fix the error in TASK [artefactual.archivematica-src : Run SS django collectstatic]

artefactual-labs/ansible-archivematica-src#126

jraddaoui · 2017-08-08T17:10:15Z

Then I'm getting the following error:

TASK [artefactual.archivematica-src : Copy shared directory structure (non RH)] ***
fatal: [am-local]: FAILED! => {"changed": true, "cmd": ["rsync", "-a", "--chown=archivematica:archivematica", "/vagrant/src/archivematica/src/MCPServer/share/sharedDirectoryStructure/", "/var/archivematica/sharedDirectory/"], "delta": "0:00:00.147970", "end": "2017-08-08 16:59:31.007168", "failed": true, "rc": 23, "start": "2017-08-08 16:59:30.859198", "stderr": "rsync: change_dir \"/vagrant/src/archivematica/src/MCPServer/share/sharedDirectoryStructure\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0]", "stderr_lines": ["rsync: change_dir \"/vagrant/src/archivematica/src/MCPServer/share/sharedDirectoryStructure\" failed: No such file or directory (2)", "rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0]"], "stdout": "", "stdout_lines": []}

Which looks to be related to artefactual/archivematica@5d1e512

jhsimpson · 2017-08-08T17:24:25Z

@jraddaoui those errors are resolved in artefactual-labs/ansible-archivematica-src#123 which just got merged

jraddaoui · 2017-08-08T17:26:45Z

Thanks @jhsimpson !

minusdavid mentioned this issue Aug 7, 2017

Support vagrant on windows #39

Closed

minusdavid mentioned this issue Aug 8, 2017

MCPClient error check Gearman worker creation artefactual/archivematica#674

Closed

sevein mentioned this issue Aug 8, 2017

Change head branch in repository from stable/1.6.x to qa/1.x (or master) artefactual/archivematica#700

Closed

minusdavid mentioned this issue Aug 10, 2017

Problem: using symlinks breaks Windows dev environments artefactual/archivematica#704

Closed

scollazo mentioned this issue Jan 31, 2018

Update to stable/1.7.x #52

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to provision using archivematica playbook with vars-singlenode-qa.yml (on Windows) #40

Unable to provision using archivematica playbook with vars-singlenode-qa.yml (on Windows) #40

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

jraddaoui commented Aug 8, 2017

jraddaoui commented Aug 8, 2017 •

edited

Loading

jraddaoui commented Aug 8, 2017

jhsimpson commented Aug 8, 2017

jraddaoui commented Aug 8, 2017

Unable to provision using archivematica playbook with vars-singlenode-qa.yml (on Windows) #40

Unable to provision using archivematica playbook with vars-singlenode-qa.yml (on Windows) #40

Comments

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 7, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

minusdavid commented Aug 8, 2017

jraddaoui commented Aug 8, 2017

jraddaoui commented Aug 8, 2017 • edited Loading

jraddaoui commented Aug 8, 2017

jhsimpson commented Aug 8, 2017

jraddaoui commented Aug 8, 2017

jraddaoui commented Aug 8, 2017 •

edited

Loading