Move node watcher task to cloud function

[niklabh]

Possible solution for memory leak: #267

We can move running task read write part to cloud function (https://cloud.google.com/functions). Cloud function are serverless compute instances which starts and tears down for each invokation. So memory leak issue will not arise.

Long running script will just invoke cloud function with blockIndex.

To test on local:

Start task runner/cloud function:

ARCHIVE_NODE_ENDPOINT=ws://127.0.0.1:9944 PRISMA_ENDPOINT=http://0.0.0.0:4466 PORT=4489 yarn start:runner

Start long running loop:

MAX_LAG=5 BLOCK_IDENTIFIER=aasdsdasdadssa ARCHIVE_NODE_ENDPOINT=ws://127.0.0.1:9944 TASK_RUNNER_SERVER=http://localhost:4489 PRISMA_ENDPOINT=http://0.0.0.0:4466 yarn start

long running loop will invoke cloud function with incrementing blockIndex. Cloud function will execute and tear down releasing memory. Invokation rate will be at the speed of block production.

[niklabh]

Need help with setting up cloud functions. I need permissions

[pmespresso]

ah cool! thank you!

my only question is whether this would slow the task runner down significantly or not since the cloud function needs to start up (and open then close a new Websocket connection, as well as instantiate and decorate PolkadotJS-api) on every block index.

It's worth a go, anyway.

[niklabh]

It will not slow down task runner. Cloud function are desinged in such a way that they take time only for some initial invokations after that cloud functions are considered warmed up and next invokations takes only some milliseconds.