initial work on docker containerization support #2479

FooBarQuaxx · 2018-04-20T10:56:51Z

This is the initial implementation of packaging easybuild buids into containers.

houndci-bot · 2018-04-20T10:57:03Z

easybuild/tools/containers/docker.py

+        if build_option('force'):
+            print_msg("WARNING: overwriting existing Dockerfile at %s due to --force" % dockerfile)
+        else:
+            raise EasyBuildError("Dockerfile recipe at %s already exists, not overwriting it without --force", dockerfile)


line too long (122 > 120 characters)

houndci-bot · 2018-04-20T10:57:04Z

easybuild/tools/containers/docker.py

+    dockerfile = os.path.join(cont_path, 'Dockerfile.%s' % file_label)
+    if os.path.exists(dockerfile):
+        if build_option('force'):
+            print_msg("WARNING: overwriting existing Dockerfile at %s due to --force" % dockerfile)


undefined name 'print_msg'

houndci-bot · 2018-04-20T10:57:04Z

easybuild/tools/containers/docker.py

+
+    module_naming_scheme = ActiveMNS()
+
+    ec = easyconfigs[-1]['ec']


local variable 'ec' is assigned to but never used

houndci-bot · 2018-04-20T10:57:04Z

easybuild/tools/containers/docker.py

+    res = []
+    _os_deps = reduce(operator.add, [obj['ec']['osdependencies'] for obj in easyconfigs], [])
+    for os_dep in _os_deps:
+        if isinstance(os_dep, basestring):


undefined name 'basestring'

houndci-bot · 2018-04-20T10:57:04Z

easybuild/tools/containers/docker.py

+
+def _det_os_deps(easyconfigs):
+    res = []
+    _os_deps = reduce(operator.add, [obj['ec']['osdependencies'] for obj in easyconfigs], [])


undefined name 'reduce'

houndci-bot · 2018-04-20T10:57:04Z

easybuild/tools/containers/docker.py

+        if build_option('force'):
+            print_msg("WARNING: overwriting existing Dockerfile at %s due to --force" % dockerfile)
+        else:
+            raise EasyBuildError("Dockerfile recipe at %s already exists, not overwriting it without --force", dockerfile)


line too long (122 > 120 characters)

houndci-bot · 2018-04-20T10:57:04Z

easybuild/tools/containers/docker.py

+    dockerfile = os.path.join(cont_path, 'Dockerfile.%s' % file_label)
+    if os.path.exists(dockerfile):
+        if build_option('force'):
+            print_msg("WARNING: overwriting existing Dockerfile at %s due to --force" % dockerfile)


undefined name 'print_msg'

houndci-bot · 2018-04-20T10:57:04Z

easybuild/tools/containers/docker.py

+
+    module_naming_scheme = ActiveMNS()
+
+    ec = easyconfigs[-1]['ec']


local variable 'ec' is assigned to but never used

houndci-bot · 2018-04-20T10:57:05Z

easybuild/tools/containers/docker.py

+    res = []
+    _os_deps = reduce(operator.add, [obj['ec']['osdependencies'] for obj in easyconfigs], [])
+    for os_dep in _os_deps:
+        if isinstance(os_dep, basestring):


undefined name 'basestring'

houndci-bot · 2018-04-20T10:57:05Z

easybuild/tools/containers/docker.py

+
+def _det_os_deps(easyconfigs):
+    res = []
+    _os_deps = reduce(operator.add, [obj['ec']['osdependencies'] for obj in easyconfigs], [])


undefined name 'reduce'

boegel · 2018-04-25T20:12:08Z

setup.cfg

+# cfr. https://stackoverflow.com/questions/47427916/how-to-config-hound-ci-to-support-python2-7
+builtins =
+    basestring,
+    reduce


@FooBarQuaxx Why add this here too, only belongs in .flake8.ini?

Sorry for the late response. setup.cfg according to the documentation is one the default locations that flake8 checks for configuration. I use flake8 as part of the development workflow ( without specifying the exact location of the config file) so I've replicated that section from .flake8.ini into the setup.cfg. Since duplication is obviously not a good thing, I can (1) keep this change (since it's more standard) and remove the reference to .flake8.ini from the houndci config files or (2) keep using the current way.

What happens if we change .hound.yml to use setup.cfg as config_file rather than using .flake8.ini?

If that doesn't cause trouble, let's make that change (maybe in a separate PR) and drop .flake8.ini?

I've made that change in #2511, together with the pure stylistic changes in easybuild/tools/options.py

Once that's merged, please re-sync this with develop

boegel · 2018-04-25T20:13:09Z

@FooBarQuaxx This needs another sync with develop, after merge of #2483

boegel · 2018-05-24T18:53:56Z

I gave this a quick look, and I think it needs a bit of work before it can go into develop. My biggest suggestions would be to:

add missing docstrings & license headers
try and avoid the hardcoding being done (e.g. Ubuntu/CentOS versions)
refactor the code to get rid of code duplication, both:
- within docker.py (the template constants for example)
- between docker.py and singularity.py; we should probably implement more general functions in containers/common.py which can then be leveraged from docker.py and singularity.py

After (finally) taking a closer look at this, I really feel this should go in first before we start tackling other issues to enhance the current containers support we have (see containers ).

@FooBarQuaxx Any idea if and when you would have time to work with us on that?
If you won't have time in the coming weeks that's fine, just let us know so we can build on top of this PR to get things in a state that we're happy with it and get it merged into develop.

FooBarQuaxx · 2018-05-24T22:25:42Z

I for sure can help on this but I want to get the clear direction. If the general consensus here is to make use of NVIDIA/hpc-container-maker then no need to continue this effort and cut to the chase so to speak by just reimplement the whole thing with that. If you want to further work on the current PR, then I guess we can sort out the relatively simple stuff (eg. headers and licensing) but leave the refactoring part to a later (near in time) stage/PR.

FooBarQuaxx · 2018-05-25T11:35:49Z

It seems some jobs are failing because of an unavailable python package.

boegel · 2018-05-25T12:40:11Z

@FooBarQuaxx I've retriggered the tests, they were failing because some kind of caching issue (we released a new version of vsc-install very recently, which seemed to be the cause of the problems).

FooBarQuaxx · 2018-06-01T07:36:54Z

@boegel I've done some refactoring according to your review on a branch other that this PR's branch. Please give feedback on whether the layout of the code is any better. Note that I intentional left out tests and code-style-validation/docstring/headers. I didn't wanted to bother creating those before getting your opinion about the shape of new code. The commit is here and the branch name on my fork is named docker-packaging-refactoring. Thanks.

boegel · 2018-06-01T20:24:11Z

@FooBarQuaxx The restructured code looks indeed more what I had in mind, and the switch to object orientation for supporting both Docker & Singularity makes a lot of sense.

There are a couple of details that I may want see changed, but overall this is a lot closer to something we can move forward with imho, thanks a lot!

So, let's update this PR accordingly and tackle the details?

boegel · 2018-06-15T20:54:46Z

@FooBarQuaxx Any updates?

FooBarQuaxx · 2018-06-16T12:22:44Z

@boegel Yes, the plan is to update this PR anytime during next week.

houndci-bot · 2018-06-18T14:25:24Z

test/framework/containers.py

+
+        self.run_main(base_args + ['--container-base=centos:7'], raise_error=True)
+
+        error_pattern = "Container recipe at %s/containers/Dockerfile.toy-0.0 already exists, not overwriting it without --force" % self.test_prefix


line too long (148 > 120 characters)

houndci-bot · 2018-06-18T14:25:25Z

easybuild/tools/containers/utils.py

+    tool_version = res.group(0)
+    version_ok = LooseVersion(str(min_tool_version)) <= LooseVersion(tool_version)
+    if version_ok:
+        print_msg("{0} version '{1}' is {2} or higher ... OK".format(tool_name.capitalize(), tool_version, min_tool_version))


line too long (125 > 120 characters)

houndci-bot · 2018-06-18T14:25:25Z

easybuild/tools/containers/singularity.py

+                print_msg("WARNING: overwriting existing container image at %s due to --force" % img_path)
+                remove_file(img_path)
+            else:
+                raise EasyBuildError("Container image already exists at %s, not overwriting it without --force", img_path)


line too long (122 > 120 characters)

houndci-bot · 2018-06-18T14:25:25Z

easybuild/tools/containers/base.py

+            if self._force:
+                print_msg("WARNING: overwriting existing container recipe at %s due to --force" % recipe_path)
+            else:
+                raise EasyBuildError("Container recipe at %s already exists, not overwriting it without --force", recipe_path)


line too long (126 > 120 characters)

houndci-bot · 2018-06-18T14:25:26Z

easybuild/tools/containers/singularity.py

+                print_msg("WARNING: overwriting existing container image at %s due to --force" % img_path)
+                remove_file(img_path)
+            else:
+                raise EasyBuildError("Container image already exists at %s, not overwriting it without --force", img_path)


line too long (122 > 120 characters)

houndci-bot · 2018-06-18T14:25:26Z

test/framework/containers.py

+
+        self.run_main(base_args + ['--container-base=centos:7'], raise_error=True)
+
+        error_pattern = "Container recipe at %s/containers/Dockerfile.toy-0.0 already exists, not overwriting it without --force" % self.test_prefix


line too long (148 > 120 characters)

houndci-bot · 2018-06-18T14:25:26Z

easybuild/tools/containers/utils.py

+    tool_version = res.group(0)
+    version_ok = LooseVersion(str(min_tool_version)) <= LooseVersion(tool_version)
+    if version_ok:
+        print_msg("{0} version '{1}' is {2} or higher ... OK".format(tool_name.capitalize(), tool_version, min_tool_version))


line too long (125 > 120 characters)

houndci-bot · 2018-06-18T14:25:26Z

easybuild/tools/containers/base.py

+            if self._force:
+                print_msg("WARNING: overwriting existing container recipe at %s due to --force" % recipe_path)
+            else:
+                raise EasyBuildError("Container recipe at %s already exists, not overwriting it without --force", recipe_path)


line too long (126 > 120 characters)

FooBarQuaxx · 2018-06-20T09:16:59Z

@boegel The tests are green now. It would be time to give any review for the structure. Afterward, I will try to add some docstring.

boegel

Overall structure looks good to me, just a bunch of minor remarks, and missing docstrings

boegel · 2018-06-26T11:13:40Z

easybuild/tools/containers/common.py

+from .docker import DockerContainer  # noqa
+from .singularity import SingularityContainer  # noqa
+
+_log = fancylogger.getLogger('tools.containers.singularity')  # pylint: disable=C0103


@FooBarQuaxx This should say tools.containers.common

boegel · 2018-06-26T11:13:52Z

easybuild/tools/containers/base.py

+from .utils import check_tool
+
+
+_log = fancylogger.getLogger('tools.containers.singularity')  # pylint: disable=C0103


use tools.containers.base but via self.log (created in __init__) rather than a global variable _log

boegel · 2018-06-26T12:28:20Z

easybuild/tools/containers/base.py

+        self._tmpdir = build_option('container_tmpdir')
+        self._container_build_image = build_option('container_build_image')
+        self._container_base = build_option('container_base')
+        self._container_path = container_path()


@FooBarQuaxx Any specific reason you're making these 'hidden' via _?

I see no need for it, and dropping the _ helps with readability imho...

Style nitpicking: please sort these initializations alphabetically

boegel · 2018-06-26T12:29:35Z

easybuild/tools/containers/base.py

+
+    def validate(self):
+        if not self._container_build_image:
+            return


Hmm, wrong place, only call validate when container_build_image is True in generate instead?

The thing here is that there are some validations that should be execute for just producing the recipe e.g. the container_base for docker should be chosen from the list of supported base docker images namely ubuntu:16.04 and centos:7.

boegel · 2018-06-26T12:31:32Z

easybuild/tools/containers/base.py

+from easybuild.tools.filetools import write_file
+from easybuild.tools.config import build_option, container_path
+from easybuild.tools.build_log import EasyBuildError, print_msg
+from .utils import check_tool


@FooBarQuaxx We don't use relative imports anywhere, so make this from easybuild.tools.containers.utils for consistency?

Also, please sort alphabetically (build_log, config, containers, filetools)

boegel · 2018-06-26T12:52:03Z

easybuild/tools/containers/utils.py

+
+def det_os_deps(easyconfigs):
+    res = set()
+    _os_deps = reduce(operator.add, [obj['ec']['osdependencies'] for obj in easyconfigs], [])


no need to name it _os_deps, since it's a local variable, os_deps is fine?

boegel · 2018-06-26T12:54:23Z

easybuild/tools/containers/utils.py

+    if not tool_path:
+        return False
+
+    print_msg("{0} tool found at {1}".format(tool_name.capitalize(), tool_path))


@FooBarQuaxx I wouldn't use capitalize here, don't see why?

I see that's not needed, but the testing code was using a capitalized string for the docker format. I can get rid of that.

Not capitalizing makes more sense to me

boegel · 2018-06-26T12:55:06Z

easybuild/tools/containers/utils.py

+
+    out, ec = run_cmd("{0} --version".format(tool_name), simple=False, trace=False, force_in_dry_run=True)
+    if ec:
+        return False


some logging would be nice here (show the error message that occurred when running --version), or even raise an error?

boegel · 2018-06-26T12:56:33Z

test/framework/containers.py

@@ -89,11 +99,11 @@ def test_parse_container_base(self):
            expected.update({'arg2': 'bar'})
            self.assertEqual(parse_container_base('%s:foo:bar' % agent), expected)

-    def run_main(self, args):
+    def run_main(self, args, raise_error=False):


@FooBarQuaxx Leave the default to True for raise_error?

boegel · 2018-06-26T12:57:38Z

test/framework/containers.py

@@ -120,6 +130,7 @@ def test_end2end_singularity_recipe(self):
        args = [
            toy_ec,
            '--containerize',
+            '--container-type=singularity',


This shouldn't be needed (since default type is Singularity)?

I thought that will be explicit and clearer.

Sure, but this way you're not testing what the default container type is, since it's always specified.

Leaving it unspecified is intentional here, it serves as a check that Singularity is the default.

boegel · 2018-06-26T14:56:51Z

easybuild/tools/containers/docker.py

+    def __init__(self, *args, **kwargs):
+        super(DockerContainer, self).__init__(*args, **kwargs)
+        # NOTE (med): set default value for _container_base
+        self._container_base = self._container_base or DEFAULT_DOCKER_BASE_IMAGE


@FooBarQuaxx Isn't this something that should be covered in tools/config.py, where we define defaults?

The current state of the --container-base option is kind of non consistent. For singularity, the option does not allow for an empty value so it does not default to anything. But for docker, we're defaulting to ubuntu:16.04. I'll change it to also be required for docker as well.

boegel · 2018-06-27T19:13:59Z

easybuild/tools/containers/base.py

+        self.easyconfigs = easyconfigs
+        self.image_format = build_option('container_image_format')
+        self.img_name = build_option('container_image_name')
+        self.log = fancylogger.getLogger(self.__class__.__module__.replace('easybuild.', ''))


please initialize the logger with fancylogger.getLogger(self.__class__.__name__, fname=False) like we do elsewhere

boegel · 2018-06-27T19:25:28Z

easybuild/tools/containers/docker.py

+    RECIPE_FILE_NAME = 'Dockerfile'
+
+    def resolve_template(self):
+        return (2 * "\n").join([


style nitpicking: I'd go with '\n\n', looks a bit weird otherwise...

boegel · 2018-06-27T19:26:14Z

easybuild/tools/containers/docker.py

+        }
+
+    def validate(self):
+        if self.container_base not in [DOCKER_BASE_IMAGE_UBUNTU, DOCKER_BASE_IMAGE_CENTOS]:


@FooBarQuaxx does it make sense to use DOCKER_OS_INSTALL_DEPS_TMPLS.keys() here?

boegel · 2018-06-27T19:27:35Z

easybuild/tools/containers/singularity.py

+
+from easybuild.tools.build_log import EasyBuildError, print_msg
+from easybuild.tools.config import CONT_IMAGE_FORMAT_EXT3, CONT_IMAGE_FORMAT_SANDBOX, CONT_IMAGE_FORMAT_SQUASHFS
+from easybuild.tools.config import container_path, build_option


style nitpicking: fix sorting here, build_option goes first

boegel · 2018-06-27T19:29:04Z

easybuild/tools/containers/utils.py

+
+    out, ec = run_cmd("{0} --version".format(tool_name), simple=False, trace=False, force_in_dry_run=True)
+    if ec:
+        print_error("command '{0} --version' failed with output: {1}".format(tool_name, out))


should be raise EasyBuildError, since print_error is not fatal?

cfr. original behaviour in check_singularity

check_tool is a predicate (returns a boolean) so raising an exception IMO is the responsibility of the caller.

@FooBarQuaxx We also raise an error when we fail to parse the version...

If running --version doesn't work, there's something seriously wrong, so the right thing to do is to exit with an error straight away I think...

boegel · 2018-06-27T19:35:44Z

test/framework/containers.py

+            "module load toy/0.0 GCC/4.9.2",
+        ]
+        self.check_regexs(regexs, def_file)
+        remove_file(os.path.join(self.test_prefix, 'containers', 'Dockerfile.toy-0.0'))


not needed strictly speaking, the test-specific temporary directory at self.test_prefix is automatically cleaned up for each test

boegel · 2018-06-27T19:37:10Z

easybuild/tools/containers/singularity.py

+
+class SingularityContainer(ContainerGenerator):
+
+    TOOLS = {'singularity': '2.4'}


@FooBarQuaxx sudo is also required here?

boegel · 2018-06-27T19:38:31Z

easybuild/tools/containers/base.py

+    def validate_tools(self):
+        for tool_name, tool_version in self.TOOLS.items():
+            if not check_tool(tool_name, tool_version):
+                raise EasyBuildError("{0} not found on your system.".format(tool_name,))


Maybe the tool is there, but it's not the correct version?

Do you suggest changing the message or being more precise about the non valid condition?

I would change the message to also include the version requirement (if there is one)

boegel · 2018-06-27T19:41:15Z

easybuild/tools/containers/docker.py

+    def resolve_template_data(self):
+        os_deps = det_os_deps(self.easyconfigs)
+
+        module_naming_scheme = ActiveMNS()


since we need this in two places, should we define a self.mns in __init__?

boegel · 2018-06-27T19:43:35Z

easybuild/tools/containers/utils.py

+from easybuild.tools.run import run_cmd
+
+
+def det_os_deps(easyconfigs):


more missing docstrings here

boegel · 2018-06-28T15:21:20Z

easybuild/tools/containers/utils.py

+    version_cmd = "{0} --version".format(tool_name)
+    out, ec = run_cmd(version_cmd, simple=False, trace=False, force_in_dry_run=True)
+    if ec:
+        print_error("command '{0}' failed with output: {1}".format(version_cmd, out))


no more need for print_error if an error is being raised?

boegel · 2018-07-05T18:03:50Z

This looks good to go, thanks a lot for all your efforts here @FooBarQuaxx!

There's certainly room for more improvement, but I don't see a need to hold this PR back any further, so I'll merge it so these changes are included in EasyBuild v3.6.2 .

Further enhancements can be done step by step, following the relevant issues in containers .

An update to the documentation (http://easybuild.readthedocs.io/en/latest/Containers.html, via https://github.com/easybuilders/easybuild/tree/master/docs) would make sense too, if you're up for it @FooBarQuaxx ...

cc @shahzebsiddiqui

FooBarQuaxx · 2018-07-05T19:47:47Z

@boegel Thanks for assisting and accepting my contribution to this awesome project.

houndci-bot reviewed Apr 20, 2018

View reviewed changes

boegel mentioned this pull request Apr 21, 2018

fix running --containerize --container-build-image with --extended-dry-run #2480

Merged

FooBarQuaxx force-pushed the docker-packaging branch from 0584a88 to 760ead4 Compare April 23, 2018 09:45

easybuilders deleted a comment from boegelbot Apr 25, 2018

boegel added this to the next release milestone Apr 25, 2018

boegel added the containers label Apr 25, 2018

boegel reviewed Apr 25, 2018

View reviewed changes

FooBarQuaxx added 6 commits May 1, 2018 14:51

initial work on docker containerization support

f948226

fixed style issues by houndci-bot

0f845c0

filter out --experimental from docker easybuild command line options

83d8132

take into account an empty os dependency value

eff5bda

added easybuild.tools.containers module to setup.py

1954435

added basic testing for docker

6112814

FooBarQuaxx force-pushed the docker-packaging branch from 5549e35 to 6112814 Compare May 1, 2018 12:51

boegel modified the milestones: 3.6.1, next release May 21, 2018

easybuilders deleted a comment from boegelbot May 24, 2018

added licensing headers and docstring

fbd4f78

easybuilders deleted a comment from boegelbot May 25, 2018

refactored code according to code reviews

6849463

fixed tests

6a581db

houndci-bot reviewed Jun 18, 2018

View reviewed changes

FooBarQuaxx added 2 commits June 18, 2018 16:43

code style fixes

c2ea677

fix failing tests

7b48a61

FooBarQuaxx force-pushed the docker-packaging branch 2 times, most recently from 7b48a61 to 3dbae48 Compare June 20, 2018 08:27

easybuilders deleted a comment from boegelbot Jun 26, 2018

boegel requested changes Jun 26, 2018

View reviewed changes

boegel reviewed Jun 26, 2018

View reviewed changes

code review changes

ed8c92e

FooBarQuaxx force-pushed the docker-packaging branch from 3dbae48 to ed8c92e Compare June 27, 2018 10:08

fixed test message

fad92b0

boegel requested changes Jun 27, 2018

View reviewed changes

more code fixes

8eea275

boegel requested changes Jun 28, 2018

View reviewed changes

FooBarQuaxx added 2 commits June 28, 2018 18:43

removed print_error

e060da8

added some missing docstrings

f16dd02

boegel approved these changes Jul 5, 2018

View reviewed changes

boegel merged commit ea3be2d into easybuilders:develop Jul 5, 2018


		module_naming_scheme = ActiveMNS()

		ec = easyconfigs[-1]['ec']


		self.run_main(base_args + ['--container-base=centos:7'], raise_error=True)

		error_pattern = "Container recipe at %s/containers/Dockerfile.toy-0.0 already exists, not overwriting it without --force" % self.test_prefix

		from .utils import check_tool


		_log = fancylogger.getLogger('tools.containers.singularity') # pylint: disable=C0103


		class SingularityContainer(ContainerGenerator):

		TOOLS = {'singularity': '2.4'}

		from easybuild.tools.run import run_cmd


		def det_os_deps(easyconfigs):

initial work on docker containerization support #2479

initial work on docker containerization support #2479

Conversation

FooBarQuaxx commented Apr 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boegel commented Apr 25, 2018

boegel commented May 24, 2018

FooBarQuaxx commented May 24, 2018 • edited Loading

FooBarQuaxx commented May 25, 2018 • edited Loading

boegel commented May 25, 2018

FooBarQuaxx commented Jun 1, 2018 • edited Loading

boegel commented Jun 1, 2018

boegel commented Jun 15, 2018

FooBarQuaxx commented Jun 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FooBarQuaxx commented Jun 20, 2018

boegel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FooBarQuaxx Jun 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FooBarQuaxx Jun 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boegel commented Jul 5, 2018

FooBarQuaxx commented Jul 5, 2018

FooBarQuaxx commented Apr 20, 2018 •

edited

Loading

FooBarQuaxx commented May 24, 2018 •

edited

Loading

FooBarQuaxx commented May 25, 2018 •

edited

Loading

FooBarQuaxx commented Jun 1, 2018 •

edited

Loading

FooBarQuaxx Jun 27, 2018 •

edited

Loading

FooBarQuaxx Jun 27, 2018 •

edited

Loading