Product Inheritance Management

[aaronjudd]

Manage product and variants through inheritance and overridden values.

When a product is cloned we add a cloneId to the product.
When a variant is cloned, we add a cloneId to the variant.

When you edit a product or variant attribute we're essential "overriding" this attribute if the object is a cloned object.

• Add functionality to reset the clones back / change to new value set in the parent hierarchy.
• Add functionality to ask change / edit only values that have not been changed in children
• Optionally override even changed/edited child values
• Update from anywhere in the hierarchy
• UI / UX to make this hierarchy visible / understandable.

The motivation to do this is for super fast mass product editing / updating.

[tdecaluwe]

Is this planned to get into the beta release? How would you implement this? Would the products be nested documents in mongodb or are they supposed to be laid out flat, querying mongo for the complete subtree when needed?

[aaronjudd]

@tdecaluwe I'm not sure if Product Inheritance Management will be in the beta - this is one of those that I'd really like to do asap though. Prior to this though I'd like us to be completely "finished" with the consumer cart->order->fulfillment flow. There's still some rough edges there, and once that's final we'll focus on the basic CMS, layout and publishing workflows.

This feature though is critical for managing any kind of larger inventory so it's probably going to need to come as part of some product page refactor. We have a rather vague issue #300 that just says "build product list view" and we're about to start addressing parts of that as we get into some layout and routing refactoring (see #502), as well as in the order fulfillment workflow @mikemurray is working on. this is related to product and publishing workflows as well, so we will probably be breaking #300 issue out into several sub tasks - somewhere in that process we need to start laying the UI out for this product management.

[aaronjudd]

@tdecaluwe the products, as well as the variants, all have a hierarchical set of ID's stored in them, referencing the product or variant that they were cloned from. We'd be querying that and build a tree from there.

[tdecaluwe]

@aaronjudd Doing subtree queries like that, can be become expensive in heavily nested trees. Instead of storing the parent it might be better to store the whole path (assuming moving queries will be used less)? Or even use a technique like nested interval trees, which is essentially the same, but coding the path as a continued fraction.

[newsiberian]

Hello, I did some changes in #212 according to latest @aaronjudd's explanations, but I faced new difficulties with variants hierarchy. Let's say what from now our cloneId prop inherited value from source._id, not from product._id.

Let's look at the cloning process:

When we cloning variants we can build hierarchy like: V1[o1, o2] <= V2[o1, o2] <= V3[o1, o2]...; We can do several clones from V2 or V3 etc. Here then we look at their cloneId's we can see some connection (relationship) between variants (top level), but if we are going to clone the entire product, we will see that there is no more such connection, because all cloneId's now point on original product variants.

Once again: when we cloning products, we lose top level variants relationships. Any ideas?

[newsiberian]

Hello. Here is an idea how we can keep hierarchy when cloning.

The simplest way without many changes, in my opinion, is to extend variants.type prop by dividing it values for variants and for child variants to something like "variant" and "child variant"/"option". This measure will allow us to add parentId to top-level variants and allow to distinguish them from child variants. At first glance, this will give us the opportunity to build nesting of any depth.

If this idea does not fit, then we can look for other ways.

[aaronjudd]

I'm working through detailed response for you now, but I do want to say that we can change the hierarchy without much issue right now. parentId is used, but I don't think anything is relying on the cloneId yet.

[aaronjudd]

Current Model

Where parentId is the ancestor in the current document, and cloneId is the ancestor from another document.

One problem with this is that each level requires an additional query in order to build the hierarchy, and because there is both a cloneId and parentId we essentially have two hierarchies with different logic.

However, neither the cloneId or parentId will fail to reproduce a hierarchy, it's just more work, more queries, than some other options.

Original product

{
    "_id" : "BCTMZ6HTxFSppJESk",
    "description" : "This is an example product. ",
    "pageTitle" : "Basic Example Product",
    "type" : "simple",
    "title" : "Example Product",
    "variants" : [
        // first variant
        {
            "_id" : "6qiqPwBkeJdtdQc4G",
            "title" : "Basic Example Variant",
            "type" : "variant"
        },
        // 1 deep - first child variant (option)
        {
            "_id" : "SMr4rhDFnYvFMtDTX",
            "title" : "Child Variant - Option 1",
            "parentId" : "6qiqPwBkeJdtdQc4G", // id of first variant
            "optionTitle" : "Option 1",
            "type" : "variant"
        },
        // 1 deep - second child variant (option)
        {
            "_id" : "CJoRBm9vRrorc9mxZ",
            "title" : "Child Variant - Option 2",
            "parentId" : "6qiqPwBkeJdtdQc4G", // id of first variant
            "optionTitle" : "Option 2",
            "type" : "variant"
        },
        // 2 deep - child child variant
        {
            "_id" : "otroLmJ2GKcyHmu9H",
            "title" : "Child Child Variant 1",
            "parentId" : "CJoRBm9vRrorc9mxZ", // id of second child variant
            "optionTitle" : "Option 2",
            "type" : "variant"
        },
        // 3 deep - child child child variant (etc)
        {
            "_id" : "h4E7TGciPByQsCaSK",
            "title" : "Child Child Variant 1",
            "parentId" : "otroLmJ2GKcyHmu9H", // id of child child variant
            "optionTitle" : "Option 2",
            "type" : "variant"
        },

        // second variant
        {
            "_id" : "6qiqPwBkeJdtdQc4G",
            "title" : "Basic Example Variant",
            "type" : "variant"
        },
        {
            "_id" : "SMr4rhDFnYvFMtDTX",
            "title" : "Basic Example Variant",
            "parentId" : "6qiqPwBkeJdtdQc4G",
            "optionTitle" : "Option 1",
            "type" : "variant"
        },
        {
            "_id" : "CJoRBm9vRrorc9mxZ",
            "title" : "Basic - Option 2",
            "parentId" : "6qiqPwBkeJdtdQc4G",
            "optionTitle" : "Option 2",
            "type" : "variant"
        }
    ],
    "workflow" : {
        "status" : "new"
    }
}

Cloned Product

{
    "_id" : "WX5Rfnt8Sitp9u4SZ",
    "description" : "This is an example cloned product. ",
    "pageTitle" : "Basic Example Product Clone",
    "type" : "simple",
    "title" : "Example Product",
    "cloneId": "BCTMZ6HTxFSppJESk" // id of first product ancestor
    "variants" : [
        // first variant
        {
            "_id" : "M9H35NXTmuSyKPtA2",
            "title" : "Basic Example Variant",
            "type" : "variant",
            "cloneId": "6qiqPwBkeJdtdQc4G", // id of first variant ancestor
        },
        // 1 deep - first child variant (option)
        {
            "_id" : "puAy4j4KnNNRpKGzW",
            "title" : "Child Variant - Option 1",
            "parentId" : "M9H35NXTmuSyKPtA2", // id of first variant
            "optionTitle" : "Option 1",
            "cloneId": "SMr4rhDFnYvFMtDTX", // id of first variant ancestor
            "type" : "variant"
        },
        // 1 deep - second child variant (option)
        {
            "_id" : "aF7GZh4k3TZHiBqpk",
            "title" : "Child Variant - Option 2",
            "parentId" : "M9H35NXTmuSyKPtA2", // id of first variant
            "optionTitle" : "Option 2",
            "type" : "variant",
            "cloneId": "CJoRBm9vRrorc9mxZ" // id of child variant ancestor
        },
        // 2 deep - child child variant
        {
            "_id" : "3amLTwhsDYue4qJYW",
            "title" : "Child Child Variant 1",
            "parentId" : "aF7GZh4k3TZHiBqpk", // id of second child variant
            "optionTitle" : "Option 2",
            "type" : "variant",
            "cloneId": "otroLmJ2GKcyHmu9H" // id of child child variant ancestor
        },
        // 3 deep - child child child variant (etc)
        {
            "_id" : "cseCBSSrJ3t8HQSNP",
            "title" : "Child Child Variant 1",
            "parentId" : "3amLTwhsDYue4qJYW", // id of child child variant
            "optionTitle" : "Option 2",
            "type" : "variant",
            "cloneId": "h4E7TGciPByQsCaSK" // id of child child child variant ancestor
        }
    ],
    "workflow" : {
        "status" : "new"
    }
}

Cloned Cloned Product

{
    "_id" : "txEyxhi8zxqShyALL",
    "description" : "This is an example cloned product. ",
    "pageTitle" : "Basic Example Product Clone",
    "type" : "simple",
    "title" : "Example Product",
    "cloneId": "WX5Rfnt8Sitp9u4SZ" // id of first product ancestor
    "variants" : [
        // first cloned variant
        {
            "_id" : "6wHAZkabcybjfbRcn",
            "title" : "Basic Example Variant",
            "type" : "variant",
            "cloneId": "M9H35NXTmuSyKPtA2", // id of first variant ancestor
        },
        // 1 deep - cloned first child variant (option)
        {
            "_id" : "2mHXJcXZPNqF6GnbA",
            "title" : "Child Variant - Option 1",
            "parentId" : "6wHAZkabcybjfbRcn", // id of cloned first variant
            "optionTitle" : "Option 1",
            "cloneId": "puAy4j4KnNNRpKGzW", // id of cloned first variant ancestor
            "type" : "variant"
        },
        // 1 deep - cloned second child variant (option)
        {
            "_id" : "eNXthmxQ3xmd3uugR",
            "title" : "Child Variant - Option 2",
            "parentId" : "2mHXJcXZPNqF6GnbA", // id of cloned first variant
            "optionTitle" : "Option 2",
            "type" : "variant",
            "cloneId": "aF7GZh4k3TZHiBqpk" // id of child cloned variant ancestor
        },
        // 2 deep - cloned child child variant
        {
            "_id" : "TY7YYdQhrGrJBmrvB",
            "title" : "Child Child Variant 1",
            "parentId" : "2mHXJcXZPNqF6GnbA", // id of cloned second child variant
            "optionTitle" : "Option 2",
            "type" : "variant",
            "cloneId": "3amLTwhsDYue4qJYW" // id of cloned child child variant ancestor
        },
        // 3 deep - cloned child child child variant (etc)
        {
            "_id" : "G7wkR6CEqQghtTFDG",
            "title" : "Child Child Variant 1",
            "parentId" : "TY7YYdQhrGrJBmrvB", // id of cloned child child variant
            "optionTitle" : "Option 2",
            "type" : "variant",
            "cloneId": "cseCBSSrJ3t8HQSNP" // id of cloned child child child variant ancestor
        }
    ],
    "workflow" : {
        "status" : "new"
    }
}

Note that we are introducing a publishing mechanism using the workflow field, so that instead of deleting
product, we'll control visibility with the workflow.status. For example, we'd have "new", "published", "unavailable" publishing status,
so that we can remove products from the active catalog, but keeping them for reference and hierarchy maintenance.
This will help keep hierarchies intact, and prevent dangling/disconnected product (to some degree).

Ancestor Model

Add ancestors array of objects to the product and variants schema. This would eliminate both parentId and cloneId, and we'd use the same logic for products and variants.

The advantage here is that you only need a single product, and you'll be able to retrieve all the parent nodes in a single query. This is presumably faster with an index on the ancestors as well.

Each time you clone, duplicate, or add either a product or variant. The ancestors will need to be updated on the new documents to reflect:

"ancestors": [
  {
    "_id": "BCTMZ6HTxFSppJESk" // original 
  },
  {
    "_id": "WX5Rfnt8Sitp9u4SZ" // first clone
  },
  {
    "_id": "BCTMZ6HTxFSppJESk" // current parent
  }
],

We'd always be cascading down the ancestor tree, rather parent tree which should make everything a little easier.

Note that while I tried to use the same ID's in all the examples, I was getting cross-eyed - so probably best to follow the comments and if the ID doesn't make sense, I'm to blame.

Cloned Cloned Product

{
    "_id" : "txEyxhi8zxqShyALL",
    "description" : "This is an example cloned product. ",
    "pageTitle" : "Basic Example Product Clone (2 deep)",
    "type" : "simple",
    "title" : "Example Product",
    "ancestors": [
      {
        "_id": "BCTMZ6HTxFSppJESk" // original product
      },
      {
        "_id": "WX5Rfnt8Sitp9u4SZ" // first clone of product
      },
    ],
    // first variant
    {
        "_id" : "M9H35NXTmuSyKPtA2",
        "title" : "Basic Example Variant",
        "type" : "variant",
        "ancestors": [
          {
            "_id": "6qiqPwBkeJdtdQc4G" // original first variant
          },
          {
            "_id": "WX5Rfnt8Sitp9u4SZ" // first clone of variant
          }
        ]
    },
    // 1 deep - first child variant (option)
    {
        "_id" : "eNXthmxQ3xmd3uugR",
        "title" : "Child Variant - Option 1",
        "optionTitle" : "Option 1",
        "type" : "variant",
        "ancestors": [
          {
            "_id": "6qiqPwBkeJdtdQc4G" // original
          },
          {
            "_id": "SMr4rhDFnYvFMtDTX" // original first child variant
          },
          {
            "_id": "M9H35NXTmuSyKPtA2" // first clone
          },
          {
            "_id": "2mHXJcXZPNqF6GnbA" // second clone
          },
          {
            "_id": "M9H35NXTmuSyKPtA2" // current parent
          }
        ]
    },
    // 2 deep - cloned child child variant
    {
        "_id" : "TY7YYdQhrGrJBmrvB",
        "title" : "Child Variant  - Option 2",
        "optionTitle" : "Option 2",
        "type" : "variant",
        "ancestors": [
          {
            "_id": "6qiqPwBkeJdtdQc4G" // original
          },
          {
            "_id": "SMr4rhDFnYvFMtDTX" // original first child variant
          },
          {
            "_id": "puAy4j4KnNNRpKGzW"  // first child variant clone
          },
          {
            "_id": "aF7GZh4k3TZHiBqpk" // second clone
          },
          {
            "_id": "M9H35NXTmuSyKPtA2" // current parent
          }
        ]
    },
    // 3 deep - cloned child child child variant (etc)
    {
        "_id" : "G7wkR6CEqQghtTFDG",
        "title" : "Child Child Variant 1",
        "optionTitle" : "Child Child Variant 1",
        "type" : "variant",
        "ancestors": [
          {
            "_id": "6qiqPwBkeJdtdQc4G" // original
          },
          {
            "_id": "CJoRBm9vRrorc9mxZ" // original first child variant
          },
          {
            "_id": "otroLmJ2GKcyHmu9H" // original first child child variant
          },
          {
            "_id": "cseCBSSrJ3t8HQSNP" // clone clone child child variant
          },
          {
            "_id": "TY7YYdQhrGrJBmrvB" // current parent
          }

        ]
    }
}

Embedded Model

It is important to note that I've always considered a variant to be a product.
A more drastic change that would be more reflective of this fact would be to use an embedded model, where the variants array is no longer a flat model, but each variant is embedded in the ancestor array.

The advantage here is that we could eliminate needing to look up the hierarchy across the product ancestors, as they would actually be embedded in each product document.

I see multiple potential disadvantages to this approach though, besides being a breaking change, and document size, the ancestor information could easily become outdated if you didn't update the entire ancestor tree anytime you changed a document.

{
    "_id" : "txEyxhi8zxqShyALL",
    "description" : "This is an example  product. ",
    "pageTitle" : "Basic Example Product",
    "type" : "simple",
    "title" : "Example Product",
    "ancestors": [
      {
        "_id": "BCTMZ6HTxFSppJESk"
      },
      {
        "_id": "WX5Rfnt8Sitp9u4SZ"
      },
    ],
    // first variant
    {
        "_id" : "M9H35NXTmuSyKPtA2",
        "title" : "Basic Example Variant",
        "type" : "variant",
        "ancestors": [
        // first variant
        {
            "_id" : "M9H35NXTmuSyKPtA2",
            "title" : "Basic Example Variant",
            "type" : "variant",
            "ancestors": [
            { // first child variant (option)
                "_id" : "puAy4j4KnNNRpKGzW",
                "title" : "Child Variant - Option 1",
                "optionTitle" : "Option 1",
                "type" : "variant",
                "ancestors": [
                  { // child child variant
                      "_id" : "G7wkR6CEqQghtTFDG",
                      "title" : "Child Child Variant 1",
                      "optionTitle" : "Child Child Variant 1",
                      "type" : "variant"
                  }
                ]
            },
            { // second child variant (option)
                "_id" : "puAy4j4KnNNRpKGzW",
                "title" : "Child Variant - Option 2",
                "optionTitle" : "Option 1",
                "type" : "variant",
                "ancestors": [
                { // child child variant
                    "_id" : "G7wkR6CEqQghtTFDG",
                    "title" : "Child Child Variant 1",
                    "optionTitle" : "Child Child Variant 1",
                    "type" : "variant"
                }
                ]
            },
            ]
        }
        ]
    }
}

[aaronjudd]

@tdecaluwe is the "Ancestor Model" I've described above similar to what you are thinking?

[aaronjudd]

Here is some more references for the Ancestor model

Mongo DB Docs
Storing Tree like Hierarchy Structures With MongoDB
MongoDB Applied Design Patterns

[tdecaluwe]

@aaronjudd Yes, that's exactly what I meant. However, the main advantage isn't to be able to get all the ancestors in one query, but all the the descendants.

I was also thinking that the Product and Variant could be merged, simplifying the interface and allowing easier querying of variants. Products in the tree can then be marked as visible, which would make them show up in the shop interface. The product page would show an overview of the whole tree, possibly through a list of selectable options (like color and size using a tree with two levels).

[aaronjudd]

I was thinking that ancestors it should work equally well for descendants, don't you think? (but yes, you'd still have to do a recursive method like we current do with parentId). Maybe you could give me an example of the structure you're thinking of?

If you added descendants to each item, the original item's descendant array could become quite large, while the children furthest in the tree would be minimal. In most cases you want (in the UI) to show the parent, then show descendants.

Products and variants are similar schemas, I could have almost called products.variants product.ancestors, and it's been suggested a few times that we should move variants into their own collection or flatten them to all be products. I think this would be a pretty large rework effort to do this, and I'm not convinced that there is a significant benefit. In my mind this is against a "document data model" where a document should be fully representative rather than needing to join/reference other items. I've leaned towards an embedded data model which provides atomicity, faster reads, and less queries than normalized data. With normalized model joins are very expensive to do, and we also lose atomicity and sharding benefits when the model is normalized (ie: joining the ancestor/descendant to via id to another document or collection).

In my mind it's better to build the document relationships by updating data on the server side and embedding in the document rather than constructing joins (either on the server or the client).

This brings me to a question of "should we just embed the variants in the ancestor array", which will make reads/queries faster, or just use a referential ID, less data stored, but more complex for referencing.

Regarding visibility, we'll be using the workflow object on the product to introduce a publishing workflow soon.

[newsiberian]

Hello. I have a few questions about these models. I noticed that these models are positioned for use in categories, not in products. When I try to understand how we will use the Ancestor model for products the worst case I see – is when somebody will clone first product, then clone this clone and so on till he get 1000+ cloned products. Each clone will have an ancestors object with 1000 – 1 records, maybe the same number in variant and in some child variant.

Is there any thoughts about this situation?

I want to understand the use cases where Ancestor model will give us speed benefits against Parent references model.

And the main question: If we will have such "heavy" product objects, how it will affect to customers expirience?

[aaronjudd]

@mikemurray and I have been testing a few different scenarios. One thing that I think we're starting to agree on is that including ancestors of the product in the data might be useless (ie, not sure if that gets us anything that search matching properties won't get us for maintenance).

Nested
http://codepen.io/mikemurray/pen/GpLWew

Flattened:
http://codepen.io/aaronjudd/pen/xwerEN

Right now I'm looking at a hybrid flattened with descendants(represented as variants in this example). This is of course, the most refactor intensive solution though, but is seemingly the most flexible of all:

    // First Top level product
    {
       "_id" : "bYggqMxT3wM62ReKP",
       "description" : "This is an example  product. ",
       "pageTitle" : "Product",
       "type" : "simple",
       "title" : "Product",
       "variants": ["KpYyQx8doBocsiHkm"]
    },
    // Second Top level product
    {
       "_id" : "txEyxhi8zxqShyALL",
       "description" : "This is an example  product. ",
       "pageTitle" : "Product",
       "type" : "simple",
       "title" : "Product",
      "variants": ["KpYyQx8doBocsiHkm", "WHgAscdQx4Z29sPwf"]
    },

    // first_variant of The first product
    {
       "_id" : "KpYyQx8doBocsiHkm",
       "title" : "Basic Example Variant - 1",
       "type" : "variant"
    },

    // first_variant of The second product
    {
        "_id" : "WHgAscdQx4Z29sPwf",
        "title" : "Basic Example Variant - 1",
        "type" : "variant",
        "variants": ["5ZPRfXp7bPRErQpQQ", "M9H35NXTmuSyKPtA2"]
    },
        // first_variant.child_variant
    {
        "_id" : "5ZPRfXp7bPRErQpQQ",
        "title" : "Basic Example Option 1",
        "type" : "variant"
    },


    // second variant of The second product
    {
        "_id" : "M9H35NXTmuSyKPtA2",
        "title" : "Basic Example Option - 2",
        "type" : "variant"
    },

Not only is this model easy to work with, but allows for sharing/reusing products. One downside is that you'll probably need to use $in, to find and filter (at least in the publication). This strategy makes it easy to have language specific variants as well. Cascading logic could be use to populate value that are empty, taken from store default locale, allowing a propagation of values

[newsiberian]

Interim results of playing with the hybrid flattened model is that there are two difficulties encountered at the very beginning:

• As we have two types of entities in one collection, we need to implement something like “multiple schemas per collection”(I have not found how to do it in mongodb) or complicate the logic of the current scheme.
• Second pain associated with subscriptions, because we need to do recursive calculation to get an array of product with all variants (or we need to subscribe to each variant separately?) in productDetail page for example. Then after adding new variants we should subscribe on then (not sure in that)? People write – the number of subscriptions is a sensitive part of the Meteor.

Any thoughts on this?

P.S. here is another one type of schema design. It calls bucket

[aaronjudd]

What I've been thinking is that we could use the "product.type" to determine if we need to use a different product variants schema. We can overwrite, or redefine the ReactionCore.Schemas.ProductVariant then based on the type. I'm not sure of the exact implementation until I work through it, but this seems like an approach that we might be able to use to keep the existing schema and methods, as one type of product then add additional methods based on the type(s).

On the subscriptions - that's a good questions, although I think it's might be more about the amount of "reactive pub/sub" not necessarily about the number of rows the subscription is monitoring. We might want format and filter the product at the publication, perhaps even do any calculations on the server side and just publish the completed product. Worth noting, that the client collection doesn't need to mirror the server collection, we can alter that within pub, or even have a client only collection that has formatted the collection. (although that makes things just hard to understand).

[tdecaluwe]

An example of what I meant with my version of the ancestor model:

// Top level product
{
  _id: 'Afg45f6dRTfe8fZi1',
  ancestors: [],
  title: 'some t-shirt',
  description: 'a general description of a t-shirt',
  visible: true,
},
// Red T-shirt
{
  _id: 'bhoFD93F56gbaqVvx',
  ancestors: ['Afg45f6dRTfe8fZi1'],
  title: 'some red t-shirt',
},
// Blue T-shirt
{
  _id: 'bhoFD93F56gbaqVvx',
  ancestors: ['Afg45f6dRTfe8fZi1'],
  title: 'some blue t-shirt',
  description: 'a more specific description mentioning cool blue t-shirt features',
},
// Blue Medium T-shirt
{
  _id: 'eW7g3Jkhp84UiAd9b',
  ancestors: ['Afg45f6dRTfe8fZi1', 'bhoFD93F56gbaqVvx'],
  title: 'some blue t-shirt - medium',
},
// Blue Large T-shirt
{
  _id: 'JGm69qr2v21uwOcnI',
  ancestors: ['Afg45f6dRTfe8fZi1', 'bhoFD93F56gbaqVvx'],
  title: 'some blue t-shirt - large',
},
// Another top level product
{
  _id: 'Hjb9d84efPks31rh7',
  ancestors: [],
  title: 'some jacket',
  description: 'a general description of a jacket',
  visible: true,
}

A separate cloneId should be used for product cloning. Similar to the embedded mode it is now possible to retrieve all data needed for the T-shirt product page in one simple query:

ReactionCore.Collection.Products.find({ ancestors._id: 'Afg45f6dRTfe8fZi1' });

While also clearly modeling the "Variants are Products" idea. We might want to use the "some blue t-shirt" product in a newsletter (linking to it's product page) while only showing the general "some t-shirt" in the online product catalog.

[aaronjudd]

@tdecaluwe @newsiberian this similar to

      "variants": ["KpYyQx8doBocsiHkm", "WHgAscdQx4Z29sPwf"]

version above? Where "variants" are descendants instead.. very similar, I guess with the same drawbacks on amount of data stored. So the question is "descendants vs ancestors"?

[tdecaluwe]

Any news on this one? It might be a good idea to choose an approach for this one so the work that's already been done can be put into a dedicated branch. Some more random reflections on the subject:

• Collections can indeed be different on the client but I don't think this should be necessary and might complicate things (as you already noted in your post). It already makes us write more code as client and server code will by definition be different.
• A product catalog still remains a 99% read/1% write problem so retrieving all variants for a product page at once is a big plus.
• Storing descendants does have a higher overhead since you store an entire tree in each node, while a list of ancestors is only a path in a tree (O(n^2) compared to O(n) for a non-degenerate tree).

In short, I think a unified schema for Products and Variants is what we need most, and I think it would work best in conjunction with an ancestors array in the schema (as described in my previous post). In any case most of the schemas would remain intact, except that they would be merged, and the variants array will be replaced.

[newsiberian]

@tdecaluwe, Hi, am I understand you correctly:

In short, I think a unified schema for Products and Variants is what we need most, and I think it would work best in conjunction with an ancestors array in the schema (as described in my previous post).

You are talking about to leave current schema without splitting on two schemas: product and variant?

As for latest changes: I did some work on the separation of schemes. You may read about it here. After that I moved all changes to another repo to this branch.

There were two main problems for me: what to do with collection2, because collection2 not ready for flattened schema, and second - what to do with publication, because it is not trivial job to get a number of products docs with all variant docs for productGrid and pagination...

If you want to test it, tell me...

P.S. I can tell about my feelings while working with flattened scheme - it is much easy for developer to work with it.

[tdecaluwe]

@newsiberian Yes, that's what I was thinking about. This also results in a flattened model, although a different one than the hybrid model. The modeled data structure is forest/a collection of trees. The main advantage should be the performance aspect. The unified schema would avoid the collection2 problems altogether. About the productGrid and pagination, what are the problems there exactly?

I didn't notice your work in reaction-catalogyet, but it's good to know things are moving forward, also I'd be happy to test/develop.

[newsiberian]

About the productGrid and pagination, what are the problems there exactly?

I don't understand yet how to call, for example - 10 product docs with all related variant docs by minimum number of db calls. For one-two call would be perfect. I do not remember exactly what was the problem, it was month ago)

I've just push latest changes with reactionImport workaround). To make all this work you will need to put this repo inside /packages and this, branch: flattened.

It would be good to hear your thoughts about this changes. Maybe this is really to much for us...

[tdecaluwe]

As for retrieving product docs with all related variants I can see the following possibilities for some different flattened models:

• Storing all descendants: Two db calls required, one for the product (with $in operator for multiple products), and another db call based on thNot suree $in operator to retrieve the variants.
• Storing only children: Similar to above but we need to recursively query the db until we have all the variants.
• Storing the ancestors: One db call matching the ancestor array against the requested products array (works like the $elemMatch operator with only one condition).

[tdecaluwe]

@newsiberian The work you've already done mostly focuses around supporting multiple schemas in one collection using meteor-collection2, am I right?

And what do you thin about the different models, specifically my proposed ancestor model? This would essentially be a materialized paths tree representation.

[newsiberian]

@tdecaluwe, hello, I overviewed your proposal about unified schema and I think we are talking about the same thing.

You suggested to store an ancestors, and this is done;
One thing we are differ in opinion is schema: You suggest ONE unified schema for product doc and for variant doc, I did two schemas for each type of docs.

Could you, please, explain why unified schema is better? How we may store unique fields like inventoryQuantity or width or pageTitle in unified schema? And the main - how should we deal with fields with defaultValue, because defaultValue means - required. Now we have fields with defaultValue in Product, but it must be absent for ProductVariant and vice versa. How this could be merged?
Maybe it will be better if you show some example with unified schema for real product fields?

The work you've already done mostly focuses around supporting multiple schemas in one collection using meteor-collection2, am I right?

In fact, client product helpers was rewritten, PDP was brought into line, products server method was also rewritten.

[tdecaluwe]

Doing some digging in the code, I see what you mean now. About the unified schema, it just seems simpler and more elegant. And as we've been saying in this thread "Variants are Products", so all product fields are meaningful for a variant. Some thoughts:

• Both width and pageTitle shoudn't pose any problems in neither schema.
• The inventoryQuantity is in the first place a variant field, but can be meaningfully defined for a products as products are essentially trees.
• I'm still thinking about defaultValue, perhaps it'd interesting to do a check on all fields with a defaultValue defined (especially for fields only defined in the ProductVariant schema)

[tdecaluwe]

@aaronjudd What do you think about unified schema vs. multiple schemas in one collection?

[newsiberian]

@tdecaluwe, I like your thoughts about unified schema, but at very beginning I was tried to put all fields in one schema with no luck. Maybe you could try to combine my changes with your idea?

Also, here is the table with current schemas (P - product, V - variant, opt - optional, defVal - defaultValue). It could be a good illustration what we have. I think we could do second table with new unified schema or two schemas:

Field	P	V	opt	defVal
_id	✅	✅
parentId		✅	✅
cloneId		✅	✅
shopId	✅
index		✅	✅
title	✅	✅	✅	✅
optionTitle		✅	✅
pageTitle	✅		✅
handle	✅		✅
description	✅		✅
metaDescription	✅		✅
barcode		✅	✅	❓
compareAtPrice		✅	✅
fulfillmentService		✅	✅
weight		✅	✅	❓
inventoryManagement		✅	✅	✅
inventoryPolicy		✅	✅	✅
lowInventoryWarningThreshold		✅	✅
inventoryQuantity		✅	✅	❓
price		✅	✅	✅
sku		✅	✅
type	✅	✅		✅
taxable		✅	✅
metafields	✅	✅	✅
createdAt	✅	✅	✅
updatedAt	✅	✅	✅
publishedAt	✅		✅
publishedScope	✅		✅
vendor	✅		✅
positions	✅		✅
variants	✅
requiresShipping	✅		✅	✅
parcel	✅		✅
hashtags	✅		✅
twitterMsg	✅		✅
facebookMsg	✅		✅
googleplusMsg	✅		✅
pinterestMsg	✅		✅
isVisible	✅			✅
workflow	✅		✅
templateSuffix	✅		✅

[tdecaluwe]

So essentially we're looking at three categories of fields, those that are in both schemas, those that are only in the Product schema and those that are only in the Variant schema:

• Fields that are in both schemas are easy, those are mostly fields like _id, createdAt and updatedAt
• Fields that are only in the Product schema are also easy, as Variants are Products. Default values for those fields don't have to be diasbled for variants. Some thoughts:
  • shopId: Do we require children to have the same shopId as their ancestors?
  • pageTitle, handle and related fields can be used on variants to upgrade them to a Product
  • description, metaDescription and similar fields should probably be inherited and can be specialized for specific variants
  • the variants field is omitted in favor of the ancestors array
• Problematic are the fields that are only in the Variant schema, since Products aren't necessarily Variants. But default values for inventoryManagement and inventoryPolicy can simply be disabled? It doesn't seem to me any of the fields which are only in the Variant schema need default values. Or am I wrong there?

[rymorgan]

Closed. We will deal with bulk editing in in #1732.