Spike: Discover and define models at runtime

[bajtos]

Certain applications don't use static JSON files to define their models, they create model definitions at runtime, often by running model discovery to obtain the model schema. Once the model is defined, the application usually exposed it via REST API.

This can be easily done in LB 3.x as follows:

const MyModel = app.registry.createModel(name, properties, settings);
app.model(MyModel, {dataSource: 'some-datasource', public: true});

Let's find out how to allow LB 4.x users to do the same.

The scope of this spike is to research solutions for discovering and defining models in such way, that the infrastructure implemented in #2036 can pick up new definitions and build REST APIs.

Acceptance criteria

• A proof-of-concept implementation of the necessary run-time parts.

• A demo application providing a new REST API endpoint that will:

  1. Accept a MySQL connection string (host+port+database+credentials) and a list of table names.
  2. Connect to the specified MySQL database.
  3. Discover model schemas from the given tables
  4. Define LB4 models for the given tables
  5. Expose these models via REST API using From model definition to REST API with no custom repository/controller classes #2036
  6. Return a list of URL paths of the newly added models (e.g. /Products, /CoffeeShops)

  We can use the public database from loopback-getting-started) when testing this new endpoint manually.

• List of GitHub issues - follow-up user stories covering the implementation. Documentation and blog post should be included in the implementation plan.

• Decide what to do about relations. The story [EPIC] From relation definition to REST API with auto-generated repository/controller classes #2483 is tracking REST API for relations, but I don't think it's a good idea to include relations in the first iteration. Create a new GH issue to add support for discovery & definition of relations, this new issue can be a spike or the real implementation, depending on whether the implementation would be clear enough.

[jannyHou]

Things to consider when do the spike:

• We probably need an open model class(weakly-typed model class) which allows any property
  • Model metadata describes the data shape of backend
  • This story is particular for relational databases(at least the first iteration let's focus on relational dbs)
  • Model instance is used in CRUD operations
• What is the run time representation of the discovered model
• Take the validation into consideration: using model definition for validation

[bajtos]

Here is a code snippet/mock-up that I wrote while discussing this feature recently:

const ds = new DataSource({/*config*/})
const discovered = await ds.connector.discover();
/* 
{
  name: 'Product',
  properties: {
    'name': {
      type: 'string'
    }
  }
}
*/

// we need to transform modelDefinitionData slightly
for (prop of discovered.properties) {
  if (Array.isArray(prop.type)) {
    prop.type = {
      type: 'array',
      itemType: prop.type[0],
    }
  }
}

// Create model definition
const modelDef = new ModelDefinition(discovered);

// Create model class
// FIXME: create a named class using a template Function factory
const modelCtor = class extends Entity {};
modelCtor.modelName = discovered.name;
modelCtor.definition = modelDef;

[bajtos]

Spike in progress: #4235

[bajtos]

Follow-up stories:

• Dynamic binding/rebinding of controllers after app start Dynamic binding/rebinding of controllers after app start #433.
• How to build models, repositories and controllers dynamically at runtime How to build models, repositories and controllers dynamically at runtime #4296
• Docs for exposing REST API of a Model with no custom classes Docs for exposing REST API of a Model with no custom classes #2740

I am closing this spike as done.

[bajtos]

Decide what to do about relations. The story #2483 is tracking REST API for relations, but I don't think it's a good idea to include relations in the first iteration. Create a new GH issue to add support for discovery & definition of relations, this new issue can be a spike or the real implementation, depending on whether the implementation would be clear enough.

I don't think we will have time to look into this in the next 3-6 months. Let's wait until there is user demand.