Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple core runnable #478

Merged
merged 23 commits into from
Sep 24, 2018
Merged

Multiple core runnable #478

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
2f6a557
add core name in config
antho1404 Sep 19, 2018
cd5ffa1
add network name based on the config
antho1404 Sep 19, 2018
a771520
add core path helper method
antho1404 Sep 19, 2018
de25479
make daemon startable multiple time with it's own name
antho1404 Sep 19, 2018
ea45586
update database path
antho1404 Sep 19, 2018
6698103
use network based on the config
antho1404 Sep 19, 2018
46a0b5c
Update container.ListServices function to accept multiple labels
NicolasMahe Sep 20, 2018
462108b
Update service.ListRunning function to only return services for a spe…
NicolasMahe Sep 20, 2018
72fdb64
Fix test
NicolasMahe Sep 20, 2018
f584cbd
update container namespace
antho1404 Sep 20, 2018
512f0e3
Merge branch 'feature/core-name-configuration' of https://github.com/…
antho1404 Sep 20, 2018
2732e0d
Change default name of Config.Core.Name from "mesg-core" to "core"
NicolasMahe Sep 21, 2018
d0d083e
Merge branch 'dev' into feature/core-name-configuration
antho1404 Sep 21, 2018
88d1d51
Merge branch 'dev' into feature/core-name-configuration
NicolasMahe Sep 21, 2018
fd91df1
Merge branch 'dev' into feature/core-name-configuration
ilgooz Sep 21, 2018
821e182
Merge branch 'dev' into feature/core-name-configuration
ilgooz Sep 24, 2018
e13071c
remove RootPath in favor of Core.Path config all the time
antho1404 Sep 24, 2018
1403c19
Merge branch 'dev' into feature/core-name-configuration
antho1404 Sep 24, 2018
7bcd8af
Merge branch 'dev' into feature/core-name-configuration
NicolasMahe Sep 24, 2018
580b0f5
fix tests
antho1404 Sep 24, 2018
8ad05bf
Merge branch 'feature/core-name-configuration' of https://github.com/…
antho1404 Sep 24, 2018
f3b67c0
dont create useless container in test
antho1404 Sep 24, 2018
ea64cae
Merge branch 'dev' into feature/core-name-configuration
antho1404 Sep 24, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,9 @@ type Config struct {
}

Core struct {
Image string
Path string
Image string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get this part, why do we need name and rootpath?
What rootpath means - consider remaining it to ConfigPath as this is what you mean right?
Also when the path is explicit given the app shouldn't create new directories (.mesg) under it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal of that is to have the database stored in it's own directory so the root path is the one that will store all the different "core" instances

Right now we have

~/.mesg/
├── database

And this change to

~/.mesg/
├── mesg-core/
|   ├── database

Like that we can now run different instances of the core with their own separated database

~/.mesg/
├── mesg-core1/
|   ├── database
├── mesg-core2/
|   ├── database
...

The RootPath is the one to store the .mesg and the CorePath() is the one to use to know where to write. This one returns the RootPath + Name.

ConfigPath is a bit ambiguous because it's the config of the core so this should be the ~/.mesg/mesg-core

Also when the path is explicit given the app shouldn't create new directories (.mesg) under it.

Why shouldn't we do that ? The goal of this property is to be the "namespace" for the folder

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the discussion on discord and the feedbacks of @krhubert we can remove the RootPath and just keep a CorePath that is customizable by the user and so we would just have the MESG_CORE_PATH to edit more

Name string
RootPath string
}

Docker struct {
Expand All @@ -62,12 +63,18 @@ func New() (*Config, error) {
c.Log.Format = "text"
c.Log.Level = "info"
c.Core.Image = "mesg/core:" + strings.Split(version.Version, " ")[0]
c.Core.Path = filepath.Join(home, ".mesg")
c.Core.Name = "core"
c.Core.RootPath = filepath.Join(home, ".mesg")
c.Docker.Core.Path = "/mesg"
c.Docker.Socket = "/var/run/docker.sock"
return &c, nil
}

// CorePath returns the path based on the Core.Name
func (c *Config) CorePath() string {
return filepath.Join(c.Core.RootPath, c.Core.Name)
}

// Global returns a singleton of a Config after loaded ENV and validate the values.
func Global() (*Config, error) {
var err error
Expand Down Expand Up @@ -100,7 +107,7 @@ func (c *Config) Load() error {

// Prepare setups local directories or any other required thing based on config
func (c *Config) Prepare() error {
return os.MkdirAll(c.Core.Path, os.FileMode(0755))
return os.MkdirAll(c.CorePath(), os.FileMode(0755))
}

// Validate checks values and return an error if any validation failed.
Expand All @@ -117,8 +124,10 @@ func (c *Config) Validate() error {
// DaemonEnv returns the needed environmental variable for the Daemon.
func (c *Config) DaemonEnv() map[string]string {
return map[string]string{
"MESG_LOG_FORMAT": c.Log.Format,
"MESG_LOG_LEVEL": c.Log.Level,
"MESG_CORE_PATH": c.Docker.Core.Path,
"MESG_SERVER_ADDRESS": c.Server.Address,
"MESG_LOG_FORMAT": c.Log.Format,
"MESG_LOG_LEVEL": c.Log.Level,
"MESG_CORE_NAME": c.Core.Name,
"MESG_CORE_ROOTPATH": c.Docker.Core.Path,
}
}
9 changes: 7 additions & 2 deletions config/config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,12 @@ func TestDefaultValue(t *testing.T) {
require.Equal(t, "localhost:50052", c.Client.Address)
require.Equal(t, "text", c.Log.Format)
require.Equal(t, "info", c.Log.Level)
require.Equal(t, filepath.Join(home, ".mesg"), c.Core.Path)
require.Equal(t, filepath.Join(home, ".mesg"), c.Core.RootPath)
require.Equal(t, "core", c.Core.Name)
require.Equal(t, "/mesg", c.Docker.Core.Path)
require.Equal(t, "/var/run/docker.sock", c.Docker.Socket)
require.True(t, strings.HasPrefix(c.Core.Image, "mesg/core:"))
require.Equal(t, filepath.Join(home, ".mesg", c.Core.Name), c.CorePath())
}

func TestGlobal(t *testing.T) {
Expand Down Expand Up @@ -77,6 +79,9 @@ func TestValidate(t *testing.T) {
func TestDaemonEnv(t *testing.T) {
c, _ := New()
env := c.DaemonEnv()
require.Equal(t, c.Log.Format, env["MESG_LOG_FORMAT"])
require.Equal(t, c.Server.Address, env["MESG_SERVER_ADDRESS"])
require.Equal(t, c.Log.Level, env["MESG_LOG_LEVEL"])
require.Equal(t, c.Log.Format, env["MESG_LOG_FORMAT"])
require.Equal(t, c.Core.Name, env["MESG_CORE_NAME"])
require.Equal(t, c.Docker.Core.Path, env["MESG_CORE_ROOTPATH"])
}
10 changes: 9 additions & 1 deletion container/container.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"github.com/docker/docker/api/types/filters"
"github.com/docker/docker/api/types/swarm"
docker "github.com/docker/docker/client"
"github.com/mesg-foundation/core/config"
)

// Container provides high level interactions with Docker API for MESG.
Expand All @@ -17,6 +18,8 @@ type Container struct {

// callTimeout is the timeout value for Docker API calls.
callTimeout time.Duration

config *config.Config
}

// Option is a configuration func for Container.
Expand All @@ -31,6 +34,11 @@ func New(options ...Option) (*Container, error) {
option(c)
}
var err error
cfg, err := config.Global()
if err != nil {
return nil, err
}
c.config = cfg
if c.client == nil {
c.client, err = docker.NewEnvClient()
if err != nil {
Expand Down Expand Up @@ -89,7 +97,7 @@ func (c *Container) FindContainer(namespace []string) (types.ContainerJSON, erro
containers, err := c.client.ContainerList(ctx, types.ContainerListOptions{
Filters: filters.NewArgs(filters.KeyValuePair{
Key: "label",
Value: "com.docker.stack.namespace=" + Namespace(namespace),
Value: "com.docker.stack.namespace=" + c.Namespace(namespace),
}),
Limit: 1,
})
Expand Down
10 changes: 6 additions & 4 deletions container/container_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,15 @@ import (
"github.com/docker/docker/api/types/filters"
"github.com/docker/docker/api/types/swarm"

"github.com/mesg-foundation/core/config"
"github.com/mesg-foundation/core/container/dockertest"
"github.com/stretchr/testify/require"
)

func TestNew(t *testing.T) {
dt := dockertest.New()
c, err := New(ClientOption(dt.Client()))
cfg, _ := config.Global()
require.Nil(t, err)
require.NotNil(t, c)

Expand All @@ -33,7 +35,7 @@ func TestNew(t *testing.T) {
require.Equal(t, "0.0.0.0:2377", (<-dt.LastSwarmInit()).Request.ListenAddr)

ln := <-dt.LastNetworkCreate()
require.Equal(t, "mesg-shared", ln.Name)
require.Equal(t, cfg.Core.Name, ln.Name)
require.Equal(t, types.NetworkCreate{
CheckDuplicate: true,
Driver: "overlay",
Expand Down Expand Up @@ -72,7 +74,7 @@ func TestFindContainerNonExistent(t *testing.T) {
require.Equal(t, types.ContainerListOptions{
Filters: filters.NewArgs(filters.KeyValuePair{
Key: "label",
Value: "com.docker.stack.namespace=" + Namespace(namespace),
Value: "com.docker.stack.namespace=" + c.Namespace(namespace),
}),
Limit: 1,
}, (<-dt.LastContainerList()).Options)
Expand Down Expand Up @@ -103,7 +105,7 @@ func TestFindContainer(t *testing.T) {
require.Equal(t, types.ContainerListOptions{
Filters: filters.NewArgs(filters.KeyValuePair{
Key: "label",
Value: "com.docker.stack.namespace=" + Namespace(namespace),
Value: "com.docker.stack.namespace=" + c.Namespace(namespace),
}),
Limit: 1,
}, (<-dt.LastContainerList()).Options)
Expand All @@ -125,7 +127,7 @@ func TestNonExistentContainerStatus(t *testing.T) {
require.Equal(t, STOPPED, status)

resp := <-dt.LastServiceInspectWithRaw()
require.Equal(t, Namespace(namespace), resp.ServiceID)
require.Equal(t, c.Namespace(namespace), resp.ServiceID)
require.Equal(t, types.ServiceInspectOptions{false}, resp.Options)
}

Expand Down
9 changes: 3 additions & 6 deletions container/namespace.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,11 @@ import (
"strings"
)

const (
namespacePrefix string = "mesg"
namespaceSeparator string = "-"
)
const namespaceSeparator string = "-"

// Namespace creates a namespace from a list of string.
func Namespace(ss []string) string {
ssWithPrefix := append([]string{namespacePrefix}, ss...)
func (c *Container) Namespace(ss []string) string {
ssWithPrefix := append([]string{c.config.Core.Name}, ss...)
namespace := strings.Join(ssWithPrefix, namespaceSeparator)
namespace = strings.Replace(namespace, " ", namespaceSeparator, -1)
return namespace
Expand Down
13 changes: 9 additions & 4 deletions container/namespace_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,20 @@ import (
"strings"
"testing"

"github.com/mesg-foundation/core/config"
"github.com/stretchr/testify/require"
)

func TestNamespace(t *testing.T) {
namespace := Namespace([]string{"test"})
require.Equal(t, namespace, strings.Join([]string{namespacePrefix, "test"}, namespaceSeparator))
cfg, _ := config.Global()
c, _ := New()
namespace := c.Namespace([]string{"test"})
require.Equal(t, namespace, strings.Join([]string{cfg.Core.Name, "test"}, namespaceSeparator))
}

func TestNamespaceReplaceSpace(t *testing.T) {
namespace := Namespace([]string{"test foo"})
require.Equal(t, namespace, strings.Join([]string{namespacePrefix, "test-foo"}, namespaceSeparator))
cfg, _ := config.Global()
c, _ := New()
namespace := c.Namespace([]string{"test foo"})
require.Equal(t, namespace, strings.Join([]string{cfg.Core.Name, "test-foo"}, namespaceSeparator))
}
4 changes: 2 additions & 2 deletions container/network.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ func (c *Container) CreateNetwork(namespace []string) (id string, err error) {
if network.ID != "" {
return network.ID, nil
}
namespaceFlat := Namespace(namespace)
namespaceFlat := c.Namespace(namespace)
ctx, cancel := context.WithTimeout(context.Background(), c.callTimeout)
defer cancel()
response, err := c.client.NetworkCreate(ctx, namespaceFlat, types.NetworkCreate{
Expand Down Expand Up @@ -69,5 +69,5 @@ func (c *Container) DeleteNetwork(namespace []string) error {
func (c *Container) FindNetwork(namespace []string) (types.NetworkResource, error) {
ctx, cancel := context.WithTimeout(context.Background(), c.callTimeout)
defer cancel()
return c.client.NetworkInspect(ctx, Namespace(namespace), types.NetworkInspectOptions{})
return c.client.NetworkInspect(ctx, c.Namespace(namespace), types.NetworkInspectOptions{})
}
10 changes: 5 additions & 5 deletions container/network_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,12 @@ func TestCreateNetwork(t *testing.T) {
require.Equal(t, id, networkID)

li := <-dt.LastNetworkCreate()
require.Equal(t, Namespace(namespace), li.Name)
require.Equal(t, c.Namespace(namespace), li.Name)
require.Equal(t, types.NetworkCreate{
CheckDuplicate: true,
Driver: "overlay",
Labels: map[string]string{
"com.docker.stack.namespace": Namespace(namespace),
"com.docker.stack.namespace": c.Namespace(namespace),
},
}, li.Options)
}
Expand All @@ -57,7 +57,7 @@ func TestCreateAlreadyExistingNetwork(t *testing.T) {
require.Equal(t, id, networkID)

li := <-dt.LastNetworkInspect()
require.Equal(t, Namespace(namespace), li.Network)
require.Equal(t, c.Namespace(namespace), li.Network)
require.Equal(t, types.NetworkInspectOptions{}, li.Options)

select {
Expand Down Expand Up @@ -88,7 +88,7 @@ func TestDeleteNetwork(t *testing.T) {
require.Nil(t, c.DeleteNetwork(namespace))

li := <-dt.LastNetworkInspect()
require.Equal(t, Namespace(namespace), li.Network)
require.Equal(t, c.Namespace(namespace), li.Network)
require.Equal(t, types.NetworkInspectOptions{}, li.Options)

require.Equal(t, id, (<-dt.LastNetworkRemove()).Network)
Expand Down Expand Up @@ -150,7 +150,7 @@ func TestFindNetwork(t *testing.T) {
require.Equal(t, id, network.ID)

li := <-dt.LastNetworkInspect()
require.Equal(t, Namespace(namespace), li.Network)
require.Equal(t, c.Namespace(namespace), li.Network)
require.Equal(t, types.NetworkInspectOptions{}, li.Options)
}

Expand Down
22 changes: 13 additions & 9 deletions container/service.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,30 +12,34 @@ import (
)

// ListServices returns existing docker services matching a specific label name.
func (c *Container) ListServices(label string) ([]swarm.Service, error) {
func (c *Container) ListServices(labels ...string) ([]swarm.Service, error) {
args := make([]filters.KeyValuePair, 0)
for _, label := range labels {
args = append(args, filters.KeyValuePair{
Key: "label",
Value: label,
})
}
ctx, cancel := context.WithTimeout(context.Background(), c.callTimeout)
defer cancel()
return c.client.ServiceList(ctx, types.ServiceListOptions{
Filters: filters.NewArgs(filters.KeyValuePair{
Key: "label",
Value: label,
}),
Filters: filters.NewArgs(args...),
})
}

// FindService returns the Docker Service or an error if not found.
func (c *Container) FindService(namespace []string) (swarm.Service, error) {
ctx, cancel := context.WithTimeout(context.Background(), c.callTimeout)
defer cancel()
service, _, err := c.client.ServiceInspectWithRaw(ctx, Namespace(namespace),
service, _, err := c.client.ServiceInspectWithRaw(ctx, c.Namespace(namespace),
types.ServiceInspectOptions{},
)
return service, err
}

// StartService starts a docker service.
func (c *Container) StartService(options ServiceOptions) (serviceID string, err error) {
service := options.toSwarmServiceSpec()
service := options.toSwarmServiceSpec(c)
ctx, cancel := context.WithTimeout(context.Background(), c.callTimeout)
defer cancel()
response, err := c.client.ServiceCreate(ctx, service, types.ServiceCreateOptions{})
Expand All @@ -53,7 +57,7 @@ func (c *Container) StopService(namespace []string) (err error) {
if err != nil {
return err
}
if err := c.client.ServiceRemove(ctx, Namespace(namespace)); err != nil && !docker.IsErrNotFound(err) {
if err := c.client.ServiceRemove(ctx, c.Namespace(namespace)); err != nil && !docker.IsErrNotFound(err) {
return err
}
timeout := 1 * time.Second
Expand All @@ -68,7 +72,7 @@ func (c *Container) StopService(namespace []string) (err error) {

// ServiceLogs returns the logs of a service.
func (c *Container) ServiceLogs(namespace []string) (io.ReadCloser, error) {
return c.client.ServiceLogs(context.Background(), Namespace(namespace),
return c.client.ServiceLogs(context.Background(), c.Namespace(namespace),
types.ContainerLogsOptions{
ShowStdout: true,
ShowStderr: true,
Expand Down
4 changes: 2 additions & 2 deletions container/service_options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ type Mount struct {
Bind bool
}

func (options *ServiceOptions) toSwarmServiceSpec() swarm.ServiceSpec {
namespace := Namespace(options.Namespace)
func (options *ServiceOptions) toSwarmServiceSpec(c *Container) swarm.ServiceSpec {
namespace := c.Namespace(options.Namespace)
return swarm.ServiceSpec{
Annotations: swarm.Annotations{
Name: namespace,
Expand Down
14 changes: 9 additions & 5 deletions container/service_options_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,9 @@ func TestServiceOptionNamespace(t *testing.T) {
options := &ServiceOptions{
Namespace: namespace,
}
expectedNamespace := Namespace(namespace)
service := options.toSwarmServiceSpec()
c, _ := New()
expectedNamespace := c.Namespace(namespace)
service := options.toSwarmServiceSpec(c)
require.Equal(t, expectedNamespace, service.Annotations.Name)
require.Equal(t, expectedNamespace, service.Annotations.Labels["com.docker.stack.namespace"])
require.Equal(t, expectedNamespace, service.TaskTemplate.ContainerSpec.Labels["com.docker.stack.namespace"])
Expand All @@ -23,7 +24,8 @@ func TestServiceOptionImage(t *testing.T) {
options := &ServiceOptions{
Image: image,
}
service := options.toSwarmServiceSpec()
c, _ := New()
service := options.toSwarmServiceSpec(c)
require.Equal(t, image, service.Annotations.Labels["com.docker.stack.image"])
require.Equal(t, image, service.TaskTemplate.ContainerSpec.Image)
}
Expand Down Expand Up @@ -52,7 +54,8 @@ func TestServiceOptionLabels(t *testing.T) {
"label2": "bar",
},
}
service := options.toSwarmServiceSpec()
c, _ := New()
service := options.toSwarmServiceSpec(c)
require.Equal(t, "foo", service.Annotations.Labels["label1"])
require.Equal(t, "bar", service.Annotations.Labels["label2"])
}
Expand Down Expand Up @@ -97,7 +100,8 @@ func TestServiceOptionEnv(t *testing.T) {
options := &ServiceOptions{
Env: []string{"env1", "env2"},
}
service := options.toSwarmServiceSpec()
c, _ := New()
service := options.toSwarmServiceSpec(c)
env := service.TaskTemplate.ContainerSpec.Env
require.Equal(t, 2, len(env))
require.Equal(t, "env1", env[0])
Expand Down
Loading