Recursive resources

[pushred]

Storyteller Editor allowed resources (and preprocessor functions) to be defined in partials, which enabled self-contained blocks that were portable, easily inserted into any page. IIRC, resources were fetched, preprocesser functions were run, and a template was rendered with the resulting context within each partial before it was inserted into the parent page. For any partials that relied on a preprocessor this could be quite slow, especially considering that a fresh V8 environment was instantiated for each preprocessor via therubyracer.

While Solidus views can be used both as pages and partials, any page configuration is ignored when views are inserted into pages as partials. If a partial has any dependencies on particular resources and anything else the context must be passed to the partial as an argument, scoped where appropriate.

For example, given this context:

{
    "resources": {
        “facebook": {
            ...
        }
    }
}

A partial that renders photos from a Facebook album would need to be inserted like this:

{{> photos resources.facebook.data }}

There’s definitely some merit to this approach, it’s evident what context partials have, and resources can be configured for the page. The downside is that if looking at the partial in isolation any API resource dependencies aren’t clear unless perhaps they are configured there for use as a page or comments are left. The resource must be setup manually regardless and can’t be updated centrally.

Reusability

The more significant downside is that this limitation doesn’t allow for the self-contained blocks we had in Storyteller. This isn’t so much of an issue within a single site, but we miss out on an opportunity with significant potential for enabling rapid development: reusing blocks across sites. Coupled with the versioning and distribution that npm/Git provide, we could reuse such blocks in the same way we do JS and CSS libraries.

Such blocks are essentially a more performant, highly customizable alternative to using iframes for such reuse. HTTP requests are kept to a minimum since the blocks depend upon assets in the parent site’s asset bundles. This also means the site’s assets can adjust CSS and interact with JS APIs as needed — iframes in contrast require serverside customization controllable via URL parameters.

Eventually Web Components will provide a better alternative altogether, but that’s still distant and doesn’t address API dependencies other than accessing them clientside. We should keep where those efforts are going in mind for our own solution however. CSS-Tricks has a good rundown of the details.

Recursive Resources

Here’s what a middleground between Storyteller Editor and Solidus might look like:

• When rendering a page, we parse the configuration found in every partial and merge the resources with that of the parent page’s. If there are any name conflicts, any parents should win.
• Since partials are currently handled by the express3-handlebars view engine we may need to take over partials, or at least parse and load any references ourselves prior to passing the template through the engine. It may be sufficiently malleable, and maybe addressed here: Intercepting Template Before Compile ericf/express-handlebars#39
• Resources defined in this “flattened” configuration would be fetched at the same time, as if they were defined in the same view.
• Preprocessors could be run in succession at this point, they should be referencing resources by name but we could run into some issues with duplication, overwritten data, etc. as the context is modified with each pass. This will have to be worked out amongst the blocks however.
• There is a question of whether to spawn child processes for each preprocessor — multiplying the processes required for each page could be a problem.
• For now, we support should be limited to 1 level deep to avoid things from getting out of hand. We should better understand the case for intra-block reusability before proceeding any further. I sense we’d be talking at an element level however, which is far more appropriate for Web Components.

[joanniclaborde]

I looked at handlebars and express3-handlebars, and there's no way to intercept partials rendering. Instead of parsing hbs files ourselves, we should use handlebars block helpers, they get called automatically by handlebars, for templates and partials. The comments block at the top of each partial would still be ignored by Solidus. A name argument can optionally be used to identify the resource in the global context.

Simple example

page1.hbs:

{{!
{
  "title": "Page 1"
}
}}

{{#resource http://site1.com}}
  {{site1key}}
  {{> page2}}
{{/resource}}

page2.hbs:

{{!
{
  "title": "Page 2"
}
}}

{{#resource http://site2.com name="site2"}}
  {{site2key}}
{{/resource}}

/page1.json:

{
  "title": "Page 1",
  "resources": {
    "http://site1.com": {"site1key": true},
    "site2": {"site2key": true}
  }
}

With nested resources

page1.hbs:

{{!
{
  "title": "Page 1"
}
}}

{{#resource http://site1.com}}
  {{#resource http://site2.com}}
    {{site1key}}
    {{site2key}}
  {{/resource}}
{{/resource}}

/page1.json:

{
  "title": "Page 1",
  "resources": {
    "http://site1.com": {"site1key": true},
    "http://site2.com": {"site2key": true}
  }
}

With a preprocessor

This new method doesn't work well with preprocessors. Nesting 3 or 4 resources, then a preprocessor helper, just looks weird. Instead, the helper could accept a preprocessor argument. That preprocessor would be responsible to load its required resources, and would only return the resulting generated context.

page1.hbs:

{{!
{
  "title": "Page 1"
}
}}

{{#resource complex.js}}
  {{complexkey}}
{{/resource}}

complex.js

var resource1 = solidus.fetchResource("http://site1.com")
var resource2 = solidus.fetchResource("http://site2.com")
return {complexkey: resource1.something + resource2.something}

/page1.json:

{
  "title": "Page 1",
  "resources": {
    "complex.js": {"complexkey": 123}
  }
}

[joanniclaborde]

Issue with this change: we'll need to fetch the resources and run the preprocessors in sync, since handlebars is not async. asyncblock is a potential solution.

[joanniclaborde]

Turns out I'm unable to make an async call sync. All the async libraries use fibers, which is fine, but as soon as an async call is made in the request, the current fiber is lost. The solution would be to use one of these libraries all over the place, and never make async calls without them. And I'm not sure it's even possible, since express3-handlebars makes async calls itself, to load the templates and partials from the file system.

I also tried handlebars-async, but I couldn't make it work, maybe it's got something to do with express3-handlebars.

The Handlebars dudes think making async helpers is a bad idea anyway, so let's find a better solution. 😠

[pushred]

Let's explore what a controller might look like. Organizing clientside code around routes is actually something else I've wanted to accomplish. Currently that's done with hackish stuff like:

if( $('body').attr('id') === 'page-index' ){
  ...
}

I've much preferred implementing a router like Crossroads.js and simply matching up code to call with the view routes derived from the filesystem. One thing I really like about this is that the parameters in the URL are all right there for reference.

I'd really like to share the same method for defining routes between the server and client and have some ability to share code as makes sense between those layers. Browserify definitely simplifies that. The question really is how to handle routing and resource fetching though. There's two projects that come to mind that have explored this with the same goal of server/client parity:

• http://ezeljs.com, read the blog post and the comment thread especially
• https://github.com/rendrjs/rendr

Backbone has been making inroads for years though in the web app world and somewhat recently in the WordPress world lately as well for more app-like presentation of content. But while it adds structure it's still very flexible and unopinionated which is great but of course translates to difficulty for newcomers. Will need some time to compose my thoughts on why our case is different or at least what layers perhaps we need to layer on top, a la Marionette.js.

Two more conventional MVC projects of note we should also look at:

• https://github.com/flatiron
• http://sailsjs.org

One fundamental difference perhaps between us and all these projects is that we're GET only. We don't need to sync data back to a server, anything session-dependent happens in the client. So we should retain flexibility to add such a layer there, while retaining a relatively simple backend. I think there's a bunch of ideas out there in terms of routing and modularity though that we should review before rolling our own.

[pushred]

We should also look at routers specifically:

• http://expressjs.com/4x/api.html#router
• https://github.com/flatiron/director
• https://github.com/alexmingoia/koa-router (mainly for reference, there are much bigger implications

One (somewhat) trivial thing I like about Crossroads.js is that it uses our {curly} syntax for parameters, which is supported in the filesystem (unlike colons) and is consistent with the URI Template syntax. All these other libraries seem to follow the :journey syntax.

Check it out in use for Storyteller.io: https://github.com/SparkartGroupInc/hipster-tools/blob/master/app/client/app/routes.js#L15. I am thinking of how our portable blocks might extend such a routing table object with a reserved base route.

[joanniclaborde]

I found express-hbs, which supports async handlebars helpers. I haven't tried to replace express3-handlebars (express-hbs has new helpers names), but I was hoping to find a solution in their codebase. Turns out each async helper returns a unique ID in sync, and they do a bunch of replace with the actual values from the helpers once the template is rendered and the async helpers are done. I gueeesss we could do something similar, but I'm not sure what would happen if for example a helper was to modify the global context...

[joanniclaborde]

I finally managed to get the resources block helper working. The trick is to run express3-handlebars' _renderTemplate (private) method inside a fiber. This is the entry point into Handlebars.compile, which is sync, so we ensure a fiber is available to make blocking async calls from the resources helper:

var Sync = require('sync')

var _renderTemplateSync = handlebars._renderTemplate
handlebars._renderTemplate = function() {
  var that = this
  var args = arguments
  Sync(function() {
    _renderTemplateSync.apply(that, args)
  })
}

handlebars.handlebars.registerHelper('resource', function(resource_url, options) {
  var resource = page.fetchResource.sync(...) // Notice the sync here
  return options.fn(resource);
});

Advantage: templates and partials can be easily reused, provided that the auth file is present between projects.

Problems: the resources are not added to the global context, since they're computer at runtime and only available during the helper's block. This means they're not included in the .json view, unless we render the view first, store the resources as we retrieve them, and finally render and send the .json view. This double-render is useless and takes time, but I guess it's acceptable if we treat the .json view as a debugging tool. This also means the resources are not included in the page's javascript content, that we inject in the page for the client code. But did we want to dump that feature anyway?

[joanniclaborde]

Another disadvantage: if multiple resources are used in a page, they will be retrieved in series, as they appear in the template. We currently request them all in parallel, so this new method is slower. And if a resource timeouts, the remaining resources will never be requested (if we maintain that 5s max render time).

[pushred]

Yeah the solution for debugging that I thought would actually work better is extending a global debugging object in the browser with the responses in each block helper. That way you could easily explore the context in your developer console. But that doesn't solve clientside resource access and 5 seconds is definitely optimistic for a % of time, especially for pages with more resources.

[joanniclaborde]

An ugly solution would be to automatically output a script tag with the resource content whenever we get them, the end result would technically be the same 😕

Also, the 5 seconds delay is arbitrary, we can increase it, but the problem is still the same: we lose some response time because we don't request all resources in parallel anymore.

[pushred]

Right that's what I'm saying.

[joanniclaborde]

I was talking about clientside resource access. Until we have a better client library.

[joanniclaborde]

BTW, is there a reason we're not using url-template instead of manually building the resource urls? We already use addressable in Hipster. They both implement RFC 6570.

Proposed setup. The idea is to have the resources and preprocessors for each template (page or partial) in modules equally accessible from the server and the client (through Browserify). Here's an example:

Server

views/index.hbs

{{# resources.news}}
  {{data}}
{{/ resources.news}}
{{> tweets}}

views/tweets.hbs

{{# resources.tweets}}
  {{data}}
{{/ resources.tweets}}

preprocessors/index.js

module.exports.resources = {
  news: storyteller('/cnn/latest')
}

module.exports.preprocessor = function(context) {
  // Do stuff to context
  return context
}

preprocessors/tweets.js

module.exports.resources = {
  api: storyteller('/twitter/tweets?page={page}')
}

module.exports.preprocessor = function(context) {
  // Do stuff to context
  return context
}

We need to trigger the right preprocessors when a page is requested. Solidus can do it magically by looking at matching file names, or we can ask the dev to explicitely wire the preprocessors with the templates. Given that we don't want magic, I see 3 solutions:

1. Add the preprocessor module name to run in each template's metadata block, like we do now (without the resources). When the server is started, each template is parsed, and each template gets assigned a preprocessor and a list of partials that will be needed.

   views/index.hbs

   {{!
   {
     "title": "Index",
     "preprocessor": "./preprocessors/index.js"
   }
   }}
   

   views/tweets.hbs

   {{!
   {
     "preprocessor": "./preprocessors/tweets.js"
   }
   }}
   

2. Create a router file, and manually load all the required preprocessors for the whole page, including the partials. The appropriate route will be called by Solidus before the page template is rendered.

   routes.js

   module.exports = function() {
     return {
       "/": function(context) {
         var index = require('./preprocessors/index.js')
         var tweets = require('./preprocessors/tweets.js'
   
         // Fetches the resources then runs all the preprocessors
         return preprocessContext(context, [index, tweets])
       }
     }
   }
   

3. Mix of 1 and 2. Solidus will call the routes for all the templates used by the page, then fetch all the resources, run the preprocessors and render the page.

   routes.js

   module.exports = function() {
     return {
       "/": function(context) {
         return require('./preprocessors/index.js')
       }
   
       // Find some way to not make this route public?
       "/tweets": function(context) {
         return require('./preprocessors/tweets.js')
       }
     }
   }
   

Client

We need to make precompiled templates available to the client, see express3-handlebars example. To share the preprocessors with the client, use Browserify. The modules usage will be similar to the server side, depending on which solution above we use.

routes.js

...
"/": function() {
  // Let's pretend there's a "Next Tweets" button
  nextTweetsElement.addEventListener('click', function (e) {
    var tweets = require('./preprocessors/tweets.js')
    var context = preprocessContext({}, [tweets], {page: currentPage + 1})

    tweetsElement.innerHTML = Handlebars.templates['tweets'](context);
  });
}

[pushred]

We can actually use this for template pre-compilation: https://github.com/epeli/node-hbsfy

Here's my take on routes.js we were looking at in our discussion:

module.exports = {

  '/index.hbs': {
    bonjovi_news: get('news.js', { 'items': 5, 'filter[tag]': 'featured' }),
    bonjovi_photos: get('photos.js', { 'items': 5, 'filter[tag]': 'featured' })
  },

  '/news.hbs': {
    bonjovi_news: get('news.js', { 'items': 15 })
  },

  '/news/{id}.hbs': {
    bonjovi_news: get('news.js')
  },

  '/blocks/tweets.hbs': {
    tweets_data: get('blocks/news.js')
  }

}

The key takeaway here is just that the params object can be extended from whatever is set at the preprocessor level through the optional arguments shown here as well as the path parameters that are defined in the filesystem.