How to build models, repositories and controllers dynamically at runtime

[bajtos]

This is a follow-up for the spike #4235

So far, our documentation and tooling was focused on the use case where most of application artifacts are static, defined in source files. In the Epic From model definition to REST API #2036, we are working on a more declarative approach, where we have a model source file and a configuration file, and the bootstrapper creates the repository & controller classes during the boot process.

In this story, I would like to show our users even more flexible and dynamic approach, where they have full control of all aspects at runtime - from defining the model class (e.g. based on schema discovery) to exposing the REST API (or decide not to).

Examples

See the demo application presented in #4235:

1. Accept a MySQL connection string (host+port+database+credentials) and a list of table names.
2. Connect to the specified MySQL database.
3. Discover model schemas from the given tables
4. Define LB4 models for the given tables
5. Expose these models via REST API
6. Return a list of URL paths of the newly added models (e.g. /Products, /CoffeeShops)

Acceptance criteria

A new documentation page with the following sections & content:

MUST HAVE

• Define a model class from a ModelDefinition object
• Define a repository class for the given model, how to configure dependencies (the datasource to use)
• Define a controller class for the given model & repository, how to configure dependencies (the repository to use) and how to build the REST API path (e.g. /products for Product model).
• Discover model definition from a database, build ModelDefinition object from the discovered schema in juggler/LB3 format.

The page can use code snippets from the demo controller (see #4235) to illustrate the approach in practice.

SHOULD HAVE

These items are not strictly required for the initial version. If you decide to leave them out, then create follow-up stories and/or move them to acceptance criteria of and existing story like #2740.

• How to use a custom Model base class (e.g. UserBase provided by the app) instead of Model or Entity.
• How to use a custom Repository base class (e.g. MyCustomRepository provided by the app) instead of DefaultCrudRepository. We need to explain how to create a custom repository class (I think this part belongs to Repositories part of our docs?) and how to build a function for creating named repository classes from the repository class. For the second part, we can tell users to copy the code from defineCrudRepositoryClass function. See also Docs for exposing REST API of a Model with no custom classes #2740.
• How to use a custom Controller to create a custom REST API. The current answer is to copy the code from defineCrudRestController function. See also Docs for exposing REST API of a Model with no custom classes #2740.

[dkrantsberg]

I came across the need to dynamically define controllers at runtime.
I was able to create and add a controller using this (somewhat hacky) approach:

    const defineNewController = new Function(`return class TestController {
        async create() {};
        async get() { return 'Hello World'; };
      }`);
     const testControllerClass = defineNewController();
    MetadataInspector.defineMetadata(
      OAI3Keys.CONTROLLER_SPEC_KEY.key,
      testControllerSpec,
      testControllerClass,
    );
    app.controller(testControllerClass);

Now is there a way to pass Request context (or request/response objects) to this controller? I need methods of this controller to be able to access request and response objects.

[dkrantsberg]

Actually I was able inject Request and Response object into a dynamically created controller by also using MetadataInspector.defineMetadata and programmatically adding injection metadata to the controller class. Like this:

const METHODS_KEY = MetadataAccessor.create<Injection, MethodDecorator>('inject:methods');

const injectionSpecs = getControllerInjectionSpecs(controllerClass);
MetadataInspector.defineMetadata<MetadataMap<Readonly<Injection>[]>>(
  METHODS_KEY,
  injectionSpecs,
  controllerClass,
);

function getControllerInjectionSpecs(target: Object): MetadataMap<Readonly<Injection>[]> {
  return {
    '': [
      {
        target,
        methodDescriptorOrParameterIndex: 0,
        bindingSelector: RestBindings.Http.REQUEST,
        metadata: {
          decorator: '@inject'
        }
      },
      {
        target,
        methodDescriptorOrParameterIndex: 1,
        bindingSelector: RestBindings.Http.RESPONSE,
        metadata: {
          decorator: '@inject'
        }
      }
    ]
  }
}  

[emonddr]

For someone that doesn't want to discover a datastructure, have an endpoint that allows the user to post a custom model definition, and continue from there.

Have a discover endpoint to which you send a connection url and it returns the discovered models.
Then POST these models into the endpoint above.

Some things to think about.

[bajtos]

Important: until #433 is implemented, controllers registered after app startup WILL NOT be picked up by our REST layer. I think we should include a note (a warning?) in the new documentation to explain the problem to readers.

[dhmlau]

@raymondfeng , I'm assigning this to you since you'll be looking into it. Thanks.

[samarpanB]

When can we expect this to be completed ? We have a project where this is required urgently. Is there a workaround right now ?
The use case we need is
We have multi-tenant architecture system with db schema based physical separation. Each tenant resides in its own DB schema. Whenever a new tenant comes in, a new DB schema will be generated and with default tables and configurations as per the defined models in codebase. There is one table which will only have a id column by default. The tenant admin user will have the capability to create new columns and setup validations for the same including default values at runtime from UI and these should persist in storage. We prefer LB4 model validations (using ajv) to take care of this, hence the need to have capability to build models, repositories and controllers at runtime via some API.

Can someone please help ?

[achrinza]

@samarpanB AFAIK dynamic (un-)binding of controllers is already possible: #4296. I'm not too sure for models and repositories.

[dhmlau]

@samarpanB, this task is part of our March milestone plan: #4767. @raymondfeng, any outlook on this? Thanks.

[raymondfeng]

@samarpanB We have already made good progress here:

In master:

• https://github.com/strongloop/loopback-next/blob/master/packages/repository/src/define-model-class.ts (defineModelClass)
• https://github.com/strongloop/loopback-next/tree/master/packages/rest-crud
• https://github.com/strongloop/loopback-next/tree/master/examples/rest-crud
• feat(rest): allow controllers to be added or removed after server start #4679

PR (optional)

• feat: move functions to define repository classes from @loopback/rest-crud to @loopback/repository #4792

What's lagging is probably a documentation page or tutorial.

[raymondfeng]

Existing docs:

• https://github.com/strongloop/loopback-next/blob/master/docs/site/Creating-CRUD-REST-apis.md (README)
• https://github.com/strongloop/loopback-next/blob/master/docs/site/Rest-Crud-generator.md
• https://github.com/strongloop/loopback-next/blob/master/examples/rest-crud/README.md (README)

I think we can convert https://github.com/strongloop/loopback-next/blob/master/docs/site/Creating-CRUD-REST-apis.md into a regular page that glues the runtime/cli/example today to tell a story.

[dhmlau]

Closing this issue as done. The documentation work will be covered in #2740.

[bajtos]

I am afraid this story was closed mistakenly.

The stories and doc pages linked in #4296 (comment) (especially #2740) are relevant for the use case where the artifacts (models, REST APIs) are static, defined by source code files (models) and configuration files (model-api config files).

However, this issue is about documenting a solution for the use case where all artifacts are dynamic, created at runtime. Quoting from the issue description:

So far, our documentation and tooling was focused on the use case where most of application artifacts are static, defined in source files. In the Epic From model definition to REST API #2036, we are working on a more declarative approach, where we have a model source file and a configuration file, and the bootstrapper creates the repository & controller classes during the boot process.

In this story, I would like to show our users even more flexible and dynamic approach, where they have full control of all aspects at runtime - from defining the model class (e.g. based on schema discovery) to exposing the REST API (or decide not to).

See also the Example section that's describing a very concrete use case that's not covered by existing documentation.

I am re-opening this issue and assigning it to 2020Q2 + May milestone.

[bajtos]

We have multi-tenant architecture system with db schema based physical separation. Each tenant resides in its own DB schema. Whenever a new tenant comes in, a new DB schema will be generated and with default tables and configurations as per the defined models in codebase. There is one table which will only have a id column by default. The tenant admin user will have the capability to create new columns and setup validations for the same including default values at runtime from UI and these should persist in storage. We prefer LB4 model validations (using ajv) to take care of this, hence the need to have capability to build models, repositories and controllers at runtime via some API.

@samarpanB I see how this feature can be useful to you. I believe all runtime bits are already in place, you can take a look at the demo application presented in #4235 to see how to use them in practice.

Recently, I have opened an issue where to discuss multi-tenancy based on dynamic schema selection - I think that's your use case too? Please join the discussion in #5056.

[bajtos]

Acceptance criteria

A new documentation page with the following sections & content:

MUST HAVE

• Define a model class from a ModelDefinition object
• Define a repository class for the given model, how to configure dependencies (the datasource to use)
• Define a controller class for the given model & repository, how to configure dependencies (the repository to use) and how to build the REST API path (e.g. /products for Product model).
• Discover model definition from a database, build ModelDefinition object from the discovered schema in juggler/LB3 format.

The page can use code snippets from the demo controller (see #4235) to illustrate the approach in practice.

I was thinking about different ways how to document this feature and more importantly, how to make such documentation easier to find by our users. Here is an alternative approach to consider:

1. Instead of creating a new doc page, update existing doc pages for DataSources, Models to describe different ways for creating the artifacts. The current "static" way via lb4 CLI, the new "dynamic" way as covered by this user story.

   Pages to update:

   • https://loopback.io/doc/en/lb4/DataSources.html
   • https://loopback.io/doc/en/lb4/Model.html
   • https://loopback.io/doc/en/lb4/Repositories.html
   • https://loopback.io/doc/en/lb4/Controllers.html

2. Convert the example app from the spike Spike: Discover and define models at runtime #4235 into a regular example app users can play with. Use example app's README to provide a high-level outline of which features the app is using and add links to new doc sections created in step 1 above to allow users to learn more details.

   There is one non-trivial aspect of such example app. The tests require a running database to discover models from. The main monorepo test suite (npm test) does not have access to databases, we have acceptance/* tests with their custom Travis CI jobs for that.

   A possible solution:

   • Keep the test inside the example app.
   • Use skipIf helper from @loopback/testlab to detect if MYSQL_HOST env variable is set. If it is not set, then skip the tests.
   • Add a new Travis CI job modeled after test-repository-mysql job, this job will setup MySQL running in a Docker and then test the example app instead of acceptance/repository-mysql.

Thoughts?

[hacksparrow]

Instead of creating a new doc page, update existing doc pages for DataSources, Models to describe different ways for creating the artifacts.

It would not be fun having to guess and search through the docs find out how to dynamically add models and controllers. However, I agree we should update those pages as well.

Convert the example app from the spike #4235 into a regular example app users can play with. Use example app's README to provide a high-level outline of which features the app is using and add links to new doc sections created in step 1 above to allow users to learn more details.

This is in theme with our current state of documentation, almost all the details are there but they are scattered around.

Let's have a single page describing how to dynamically add models and controllers. A succinct doc with all the info on the same page causes minimal context switching and brings great joy to the reader 😃

[hacksparrow]

Addressed in #5591.

[hacksparrow]

Closed via #5591 and #5630.