Mentos integration #54

Arttii · 2017-02-06T11:34:36Z

Description

Migrating satyr to new CI, Naming and underlying driver to Mentos and Daskos

Motivation and Context

A pure python high level implementation
One general solution for all related packages

How Has This Been Tested?

Tests are still pending

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

Review:

@kszucs

First assignee approved
Second assignee approved

kszucs

So far so good, just some comments. I know its under construction.

kszucs · 2017-02-07T07:08:01Z

mentor/messages.py

+        return codecs.unicode_escape_decode(string)[0]
+
+# dict.iteritems(), dict.iterkeys() is also incompatible
+if six.PY3:


iteritems and iterkeys are importable from six

from six import iteritems, iterkeys

Ya I know, I stole that block of code and was lazy

kszucs · 2017-02-07T07:09:10Z

mentor/messages.py

+            except KeyError:
+                raise AttributeError(k)
+        else:
+            object.__delattr__(self, k)


Are these for dot notation?

kszucs · 2017-02-07T07:09:57Z

mentor/messages.py

+    return Message({'name': 'disk', 'role': '*', 'scalar': Message({'value':value}), 'type': 'SCALAR'})
+
+def Ports(begin,end):
+    return Message({'name': 'disk', 'role': '*', 'ranges': Message({'range':[Message({'begin': begin, 'end': end})]}), 'type': 'SCALAR'})


Needs formatting

God yes, maybe setup autopep8 or something, I did not do anything on this package

kszucs · 2017-02-07T07:17:44Z

mentor/messages.py

+        b = self.flatten(other)
+        out = {}
+        for k, v in self.zeroes.items():
+            if k=="ports":


With supporting ports it could be cleaner to create scalar and range types and convert the resources when initializing the ResourceMixin.

kszucs · 2017-02-07T07:18:50Z

mentor/messages.py

+        return self.keys()
+
+def Cpus(value):
+    return Message({'name': 'cpus', 'role': '*', 'scalar': Message({'value':value}), 'type': 'SCALAR'})


Handling role will be an interesting question.

kszucs · 2017-02-07T07:21:44Z

mentor/messages.py

+
+    def __init__(self, docker='satyr', force_pull=False,
+                 envs={}, uris=[], **kwds):
+        super(PythonExecutor, self).__init__(**kwds)


Maybe we should use something like a template with pure dict notation instead of Message constructors.

So like the Cpus and Disk?

kszucs · 2017-02-07T07:23:00Z

mentor/placement.py

@@ -21,10 +21,8 @@ def weight(items, **kwargs):

 def ff(items, targets):
    """First-Fit
-


What is your problem with my whitespaces? :)

It be my editor I think :D

kszucs · 2017-02-07T07:24:29Z

mentor/queues.py

-        self.client = client
+    def __reduce__(self):
+        hosts = ",".join(["{}:{}".format(h, p) for h, p in self.client.hosts])
+        return (SerializableMixin.init, (self.__class__, hosts, self.path))


Cleaner indeed.

kszucs · 2017-02-07T07:29:09Z

.pre-commit-config.yaml

@@ -1,28 +0,0 @@
- repo: https://github.com/pre-commit/pre-commit-hooks.git


I like to use these hooks http://pre-commit.com/

Enforce me to keep the conventions. Do You have alternatives?

kszucs · 2017-02-07T07:31:31Z

Nice to see the process :)

kszucs · 2017-02-24T07:01:18Z

What is the status of this? I'd like to implement a framework for distributed, but kinda blocked this PR.

Arttii · 2017-02-28T08:44:35Z

Well, it needs a fairly extended rewrite I think or rethink of how we do things. I did not have time to work on this recently as well. Need to properly understand how to do it.

kszucs · 2017-02-28T08:46:07Z

Let me know if I can help.

Arttii · 2017-02-28T08:59:51Z

What do you think are the main necessities we need in mentor?

Because stuff like retry and etc are already handled in mentos, and the way it works with tornado callbacks we cannot intercept exceptions, but have to kinda propagate errors as events, meaning the function evaluation thing has to be fully redone. I am also very unsure about the function execution interface.

You can take a look at the currently state. Its's a mess i think.

kszucs · 2017-02-28T09:03:27Z

Is the like retry configurable in mentos? Can You link the source of the retry logic?

I am also very unsure about the function execution interface.

Do You have proposals?

Arttii · 2017-02-28T09:06:57Z

The retry is not directly exposed to the schedulers as of now, but can be made a global config I guess.

I am quite unsure to be honest. I keep getting caught up in making everything work with the existing codebase.

kszucs · 2017-02-28T12:47:53Z

Ohh, this retry logic is great. I didn't know that mentos has it.

The problem is that we have multiple options to place the scheduling logic: scheduler, executor, even task. We also need to distinguish between homogeneous and heterogeneous tasks.

We should specify a couple of frameworks (on pseudo code level) to plan the proper interface for mentor. We might need multiple interfaces.

kszucs · 2017-02-28T12:53:25Z

I've created a new issue #55 to plan the interface.

kszucs · 2017-03-10T07:40:10Z

I'm closing this PR now, created a new mentos branch with your modifications.

kszucs and others added 26 commits September 7, 2016 16:01

Added support for unified containerizer; updated drone configuration

34b3725

Removed garbage test file

d73b896

Small refactoring

dec6e91

Changed SerializableMixin in Queue to use __reduce__

9937f61

Changed logging level to error in drone service containers

8aff59d

Docker-registry log level to error

63c8f27

Increased resources for mesos-slave-1

98e0296

Increased timeout for a futures test

c23672b

Increased timeout for a multiprocessing test

fbac88f

New drone signed yml

6bee8da

PythonExecutor is now a standalone message

99ee0ae

CI fixes

2958e11

Fixed imports in satyr.__init__

621c397

Changed dockerfile to deban+miniconda+mesos

eb1356e

Avoid whiteout files in dockerfile

a2eb166

Honor slave attributes as contrsaints, fixes daskos#47

e338787

Fixed QueueScheduler arguments

0c9dcd4

using malefico as driver

c945fe5

Magic

d4ad986

temp

dda7260

Ehm base passing

9024366

added json test

5a312a9

refactor daskos#1

f2ae217

test suite passes

4d236f9

added ports defaults

5f9faac

preparing for pull request

b1045a2

Arttii requested a review from kszucs February 6, 2017 11:36

Arttii self-assigned this Feb 6, 2017

Arttii added enhancement help wanted labels Feb 6, 2017

kszucs reviewed Feb 7, 2017

View reviewed changes

some more progress

e8a9fa1

need a redo

3d5e9eb

Merge branch 'master' into mentos

67de2be

kszucs changed the base branch from master to mentos March 10, 2017 07:39

kszucs closed this Mar 10, 2017

kszucs reopened this Mar 10, 2017

kszucs merged commit 362d38c into daskos:mentos Mar 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mentos integration #54

Mentos integration #54

Arttii commented Feb 6, 2017

kszucs left a comment

kszucs Feb 7, 2017

Arttii Feb 7, 2017

kszucs Feb 7, 2017

Arttii Feb 7, 2017

kszucs Feb 7, 2017

Arttii Feb 7, 2017

kszucs Feb 7, 2017

kszucs Feb 7, 2017

kszucs Feb 7, 2017

Arttii Feb 7, 2017

kszucs Feb 7, 2017

Arttii Feb 7, 2017

kszucs Feb 7, 2017

kszucs Feb 7, 2017

kszucs commented Feb 7, 2017

kszucs commented Feb 24, 2017

Arttii commented Feb 28, 2017

kszucs commented Feb 28, 2017

Arttii commented Feb 28, 2017

kszucs commented Feb 28, 2017 •

edited

Loading

Arttii commented Feb 28, 2017 •

edited

Loading

kszucs commented Feb 28, 2017

kszucs commented Feb 28, 2017

kszucs commented Mar 10, 2017

		@@ -21,10 +21,8 @@ def weight(items, **kwargs):

		def ff(items, targets):
		"""First-Fit

		@@ -1,28 +0,0 @@
		- repo: https://github.com/pre-commit/pre-commit-hooks.git

Mentos integration #54

Mentos integration #54

Conversation

Arttii commented Feb 6, 2017

Description

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Review:

kszucs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs commented Feb 7, 2017

kszucs commented Feb 24, 2017

Arttii commented Feb 28, 2017

kszucs commented Feb 28, 2017

Arttii commented Feb 28, 2017

kszucs commented Feb 28, 2017 • edited Loading

Arttii commented Feb 28, 2017 • edited Loading

kszucs commented Feb 28, 2017

kszucs commented Feb 28, 2017

kszucs commented Mar 10, 2017

kszucs commented Feb 28, 2017 •

edited

Loading

Arttii commented Feb 28, 2017 •

edited

Loading