Integration of the orchestrator API and authentication based on payload signature #221

antho1404 · 2020-04-05T08:58:21Z

Add new .mesgrc config to store user's mnemonic and config for the engine
Add user's public key to the engines authorized accounts
Create orchestrator library based on the proto definition
Use orchestrator lib for the runner
Use orchestrator lib in the CLI
Sign all requests from the CLI with user's mnemonic
Add token to the user from the engine's account

Test

Use the branch test/cli-mnemonic that includes a runner that doesn't run in docker (and has access to the library not yet released)
Build the engine image make docker-dev
Select a service
npm install path-to-the-js-sdk/packages/service
`./bin/run service:dev path-to-service --version local

github-actions · 2020-04-06T05:39:21Z

Please update the CHANGELOG of the associated library for your Pull Request to be accepted

antho1404 · 2020-04-08T15:00:30Z

packages/cli/src/commands/service/execute.ts

          })
        }
      }
    ])
    await tasks.run()

-    cli.styledJSON(decode(execution.outputs))
+    styledJSON(decode(execution.outputs))
+    process.exit(0)


This is to exit the process without having to cancel the log and have the warning from GRPC.

it will be nicer to catch the error and not displaying it in order to really remove the stream from the server.
But ok as it's local dev only

antho1404 · 2020-04-08T15:01:11Z

packages/cli/src/utils/config.ts

+// https://github.com/forbole/big-dipper/blob/master/imports/startup/server/util.js
+const pubKey = (mnemonic: string): string => {
+  const publicKey = Account.deriveMnemonic(mnemonic).publicKey
+  const pubkeyAminoPrefix = Buffer.from('EB5AE98721', 'hex')


More information here and here

packages/cli/src/utils/environment-tasks.ts

antho1404 · 2020-04-08T15:05:13Z

packages/orchestrator/src/event.ts

+    super(endpoint, 'Event')
+  }
+
+  public stream(request: StreamRequest, signature: string): grpc.ClientReadableStream<Type.mesg.types.IEvent> {


StreamRequest and all attributes of these functions are here in order to "normalize" the data between the serialization in json/amino and the one in proto.
The signature is based on the json serialization while the payload is sent in proto which is really confusing.

antho1404 · 2020-04-08T15:07:27Z

packages/orchestrator/src/execution.ts

+
+// TODO: To improve using the types but if include the type, the definition is still included in the js
+export const Status: { [key: string]: Type.mesg.types.Status } = {
+  Unknown: 0, // Type.mesg.types.Status.Unknown,


Read the todo, there is some weird behavior with enums in typescript, the file stays included when compiled even if it doesn't generate any js... maybe a bug in typescript

packages/api/src/transaction.ts

NicolasMahe · 2020-04-09T04:34:15Z

packages/cli/src/commands/service/execute.ts

          })
        }
      }
    ])
    await tasks.run()

-    cli.styledJSON(decode(execution.outputs))
+    styledJSON(decode(execution.outputs))
+    process.exit(0)


it will be nicer to catch the error and not displaying it in order to really remove the stream from the server.
But ok as it's local dev only

packages/cli/src/utils/environment-tasks.ts

NicolasMahe · 2020-04-09T04:38:31Z

packages/cli/src/utils/environment-tasks.ts

+        const userBalance = userAccount.coins.find(x => x.denom === 'atto')
+        if (userBalance && parseInt(userBalance.amount, 10) > 1000) return
+        const transferMsg = api.account.transferMsg(engineAccount.address, userAccount.address, [{
+          amount: 1e18.toString(),


is it enough? 1e18 is 1 actual coin.
in the engine e2e, it transfers 100,000,000 x 10^18 atto

Can definitely be increased, just here the engine will do all the processing, the CLI should just use that to deploy. So the question is is it enough to deploy multiple processes and services.
I think 100k is really a lot just for deployment

packages/service/CHANGELOG.md

packages/service/src/index.ts

Co-Authored-By: Nicolas Mahé <[email protected]>

NicolasMahe

https://github.com/mesg-foundation/js-sdk/pull/221/files#diff-7061e640ee22641a8cbd0d23cd1903efR135

const logs = res.split('\n').map(x => {
  try {
    return JSON.parse(x)
  } catch (e) {
-    return null
+    return reject(e)
  }
}).filter(x => x)

antho1404 mentioned this pull request Apr 5, 2020

Add Orchestrator gRPC API mesg-foundation/engine#1771

Merged

antho1404 added 2 commits April 6, 2020 12:30

improve configuration management

b5161d6

authorize cli for the engine's admin api

eeb4d46

antho1404 force-pushed the feature/cli-mnemonic branch from 629bc51 to eeb4d46 Compare April 6, 2020 05:39

update registration of the service

c95ff06

antho1404 changed the title ~~Feature/cli mnemonic~~ orchestrator api Apr 7, 2020

antho1404 added 21 commits April 7, 2020 20:05

update changelog

468dc38

fix sign return params in transaction

4156090

create orchestrator lib

b14dc3d

use signature instead of private key

062910b

use @mesg/orchestrator library for @mesg/service

136a88a

use direct proto messages for orchestrator

2e1f7be

update service from orchestrator

739c0fe

fix types

527d46d

normalize payload for the orchestrator

b044d39

add token to the user

dd35c70

use orchestrator for streams in the cli

5c8f494

update service based on the orchestrator type

ed306cd

calculate runner hash based on engine address

2ba6b2e

update runner to delete when stop based on the orchestrator

cb3ac89

add missing proto file

1852bae

fix missing dep

dca1e95

update fetch of proto

242b554

generate pub key from mnemonic for authorized clients

15e0f1b

add lcd struct type with convertion

56fe63e

update execution command based on orchestrator

7146126

cleanup

5fa8958

antho1404 commented Apr 8, 2020

View reviewed changes

antho1404 marked this pull request as ready for review April 8, 2020 15:10

antho1404 changed the title ~~orchestrator api~~ Integration of the orchestrator API and authentication based on payload signature Apr 8, 2020

NicolasMahe reviewed Apr 9, 2020

View reviewed changes

antho1404 and others added 2 commits April 9, 2020 11:50

update changelogs

0e067fe

Apply suggestions from code review

af80055

Co-Authored-By: Nicolas Mahé <[email protected]>

NicolasMahe requested changes Apr 9, 2020

View reviewed changes

antho1404 added 3 commits April 9, 2020 13:43

fix pending promises

3115b76

remove api dependency in container

2c985cc

Merge branch 'master' into feature/cli-mnemonic

03ad0c2

NicolasMahe approved these changes Apr 9, 2020

View reviewed changes

NicolasMahe assigned antho1404 Apr 9, 2020

NicolasMahe added this to the next milestone Apr 9, 2020

antho1404 merged commit 30d8454 into master Apr 9, 2020

antho1404 deleted the feature/cli-mnemonic branch April 9, 2020 07:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration of the orchestrator API and authentication based on payload signature #221

Integration of the orchestrator API and authentication based on payload signature #221

antho1404 commented Apr 5, 2020 •

edited

Loading

github-actions bot commented Apr 6, 2020

antho1404 Apr 8, 2020

NicolasMahe Apr 9, 2020

antho1404 Apr 8, 2020

antho1404 Apr 8, 2020

antho1404 Apr 8, 2020

NicolasMahe Apr 9, 2020

NicolasMahe Apr 9, 2020

antho1404 Apr 9, 2020

NicolasMahe left a comment

Integration of the orchestrator API and authentication based on payload signature #221

Integration of the orchestrator API and authentication based on payload signature #221

Conversation

antho1404 commented Apr 5, 2020 • edited Loading

Test

github-actions bot commented Apr 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NicolasMahe left a comment

Choose a reason for hiding this comment

antho1404 commented Apr 5, 2020 •

edited

Loading