[p5.js 2.0] State machines and renderer refactoring

[limzykenneth]

Renderer state machine

The base p5.Renderer class will provide a states property that will keep track of all states within the renderer that are affected by push() and pop(). The base version looks like the following:

this.states = {
      doStroke: true,
      strokeSet: false,
      doFill: true,
      fillSet: false,
      tint: null,
      imageMode: constants.CORNER,
      rectMode: constants.CORNER,
      ellipseMode: constants.CENTER,
      textFont: 'sans-serif',
      textLeading: 15,
      leadingSet: false,
      textSize: 12,
      textAlign: constants.LEFT,
      textBaseline: constants.BASELINE,
      textStyle: constants.NORMAL,
      textWrap: constants.WORD
};

Each renderer that extends p5.Renderer should add any additional states to the states properties when it should be affected by push() and pop(), eg. this.states.someState = 'someValue'. The base implementation of push() and pop() will keep track of the states with push() pushing a copy of the current state into an array then pop() restores the last saved state in the array to the renderer's current state. (See line comments for a bit more details below)

The individual renderer should call super.push() and super.pop() in their own implementation of push() and pop() in addition to acting on the necessary changes in states.

OOP hierachy

The previous hierachy has p5.Element as the base class which p5.Renderer extends and p5.Renderer2D/GL again extends. This means all renderer are assumed to have a p5.Element backing. Part of the changes made here is to remove this assumption so p5.Renderer can be extended into renderers that does not render to a HTML element.

Eventually, the individual renderers p5.Renderer2D and p5.RendererGL will have their own reference to p5.Element that they can operate through.

Overall event handling is still something that needs to be think through for this case though.

Global mode

_setProperty() is no longer used. Global variables will now all be using getters that refers to the value in the p5 instance.

Pending global functions to be attached in the same way.

[limzykenneth]

This should probably be using structuredClone() because states may have objects as properties that won't be fully preserved if copied in this way. It is currently not structuredClone() because somewhere in the WebGL renderer states is a reference to window which cannot be cloned.

[davepagurek]

I believe only some base types work in structured clone, and I'm not certain all the things we put into this state conform to that (e.g. in WebGL, currently we have a camera object.) We have a few options here, do you have any preference between:

• making an interface like clone() that state properties can optionally implement so that we can have custom object types in the state
• forcing all state to be a base type and not storing custom classes

I think the former might be a smaller refactor, but that extra flexibility might also mean a higher surface area for bugs. If we go for the latter, for WebGL, I think that would mean storing camera properties rather than the camera itself, and adding a custom handler on pop() to reapply those properties to the active camera.

[davepagurek]

I think for performance we might want the option of not deep copying everything if it isn't necessary, so I maybe slightly lean towards the former?

[limzykenneth]

Some of the key things not cloneable for our use will be (including nested in objects) symbols, functions, and DOM nodes.

making an interface like clone() that state properties can optionally implement so that we can have custom object types in the state

Does this mean we will loop through each direct properties of the state object and if there is a method clone(), call it and store that?

forcing all state to be a base type and not storing custom classes

I'm slightly more in favor of this and I think it is similar to the above option (if I understand it correctly) but with the renderer itself doing the clone() step before setting it in the state object.

For something like p5.Camera it was never suited for deep cloning anyway as it would be duplicating the underlying p5 instance and what not. Saving the camera properties then reapplying them to the active camera feels more semantic but I'm not sure if there will be any performance impact (through repeated instantiation of heavy object for example)?

Another option is that the base p5.Renderer.states object is essentially a convenience in that the push and pop stack will be implemented already in the base class so the derived class don't need to implement it, but if the derived class needs more complex behavior including using non-structuredCloneable states, it can perhaps be handled independently in the derived class's push() pop() implementation, if that made sense.

[davepagurek]

Does this mean we will loop through each direct properties of the state object and if there is a method clone(), call it and store that?

Yep!

Another option is that the base p5.Renderer.states object is essentially a convenience in that the push and pop stack will be implemented already in the base class so the derived class don't need to implement it, but if the derived class needs more complex behavior including using non-structuredCloneable states, it can perhaps be handled independently in the derived class's push() pop() implementation, if that made sense.

My comment about perf was mostly about avoiding automatic deep copying of everything, so I'm good with this too if the more expensive clones are done explicitly when needed. So maybe this method would mean that by default, everything is a simple type that should be replaced instead of mutated in place so we can keep doing Object.assign(...) instead of a structuredClone(...), and then a renderer can do whatever copy it needs manually in its push override? And it should be just a few things needing that custom behaviour.

[limzykenneth]

If we can further limit state properties to be basically primitives only (copy types only), Object.assign() should be enough, it is just when the state properties are objects or arrays then structuredClone() would be needed. I'm fine going with the primitives only approach for an overall lighter state management by default if there's no obvious need for reference types to be used as state properties?

If that sounds good, I can try to have a look at WebGL renderer states or since there are other WebGL stuff going on as well, maybe you'd have a better idea of when to review them? I'm going to move back onto the hierachy refactor and instance states next so it's not immediately blocking at this point.

[davepagurek]

hmm right, I guess we have a lot of array states currently in WebGL as that's the format colors are stored in, and those are all treated as reference types right now. To avoid deep copies on all those, we'd have to wrap them in some other class to avoid the structured clone, so it feels like maybe that'd add more overhead than it's worth. Are there many 2D properties that need a clone? Would relying on object reference for everything by default be a possibility?

I tried doing some quick find-and-replacing this morning to start converting state into WebGL here: https://github.com/limzykenneth/p5.js/compare/2.0-modules...davepagurek:p5.js:2.0-webgl?expand=1 This is currently using the .copy() method on things if it exists, but we can change how that's done based on what we decide here, and currently this changeover has broken more tests that will need looking into. But anyway since this is just changing how some properties are accessed, I think it's ok to make these changes whenever, and it won't be too big of a merge for the other contributors.

[limzykenneth]

For 2D the only one using reference type is tint using an array of colors as far as I remember, but it is actually currently on the base p5.Renderer class as WebGL can tint as well. If there is either a guarantee or just enforcement through documentation that if reference types are set as state properties, it must not be separately modified, eg. additional tints creates new array and never reuses it, it would still be fine to just Object.assign().

You should be able to push to this PR directly if you want to do the conversion and test fixing as necessary. I'll merge from dev-2.0 regularly as I work on this too.

[limzykenneth]

@davepagurek Following on what we discussed regarding the renderer, I've set up an implementation of the renderer state machine here. For 2D most of the things are working since it is relatively simple, for WebGL however it seems there are some states not being tracked correctly, causing tests to fail.

[limzykenneth]

For changing the hierachy of the OOP structure, p5.Element is created as a property wrappedElt of p5.Renderer2D as p5.Renderer may not need any DOM nodes. This creates a problem which is that p5.Element methods are expected to be present in the renderer object (returned by createCanvas() for example) so that one may call .id() or .mousePressed() on it.

This change above more or less preserve the existing behavior outwardly by aliasing p5.Renderer.id() to p5.Renderer.wrappedElt.id() using getter (may not be necessary and can just be directly assigned). This gives p5.Renderer2D the appearance of being p5.Element but not technically an instance of it.

Another less magical option is to make a breaking change so that the renderer's underlying p5.Element can only be accessed through a property of the renderer, and so getting the id of the element backing a renderer would be more like canvas.element.id() instead of the current canvas.id(). This option has potential to break quite a bit of existing sketches though as it is quite common to do canvas.parent("#some-div") to place a canvas in the DOM.

[davepagurek]

I'm ok with either of those (maybe we could make a mixin() function so that other renderers can easily do the same thing if we go with the first option?)

Another thought: do users expect a renderer to come out of createCanvas or could that return the element of the renderer? Off the top of my head I can't think of (public) APIs on a renderer itself that aren't just on p5.Element that people might be using. I suppose not all renderers have an element to return though (maybe undefined is a valid return type here?)

[limzykenneth]

Hmm...perhaps returning p5.Element for those using it could be something createCanvas returns as it aligns with the other createX() functions in the DOM module as well. However in this case I guess the created renderer will not be available publicly necessarily then.

[davepagurek]

Right, I think that's OK if renderers are private if renderers have a way of adding public methods to p5. So things like sphere() could exist just in the WebGL renderer, added to p5

[limzykenneth]

Yeah, that one is somewhere on my list for this PR, might go with the existing method (defining p5.prototype.sphere() that calls this._renderer.sphere() or might go with the getter technique here as well. We can discuss when I get to that implementation.

[davepagurek]

oops I resolved the merge conflicts with the dev-2.0 updates, but that also brought in some more bits that I need to convert to use states. I'll push another commit soon

[limzykenneth]

@davepagurek Probably shouldn't update right away but just to let you know in advanced. The refactor for Renderer2D moves all DOM related operation and references to be handled by the renderer's createCanvas method along with reshuffling of the initialization logic around default canvas and others. As such WebGL renderer will also need updating accordingly.

I'm working on getting p5.Graphics to work first before I do a round of clean up refactoring.