Metadata picking preparation

[javagl]

The 3D Tiles Next Metadata Compatibility Matrix currently lists the functionalities of picking and styling based on property textures or property attributes as "not supported". Some details about this requirement are also summarized in #9852

This PR is supposed to help fleshing out possible approaches for this. I have not worked with the relevant parts of the code. And the compatibility matrix contains a footnote saying

Picking/styling property textures will require significant refactoring of CesiumJS' picking system.

but I have no idea what this really entails. So I'm opening this as an early DRAFT, to get an idea about how flawed the approach is, and gather feedback for "better" approaches.

---

30-second-summary

The naive description of the approach is:

• Add a function like Scene.pickMetadata(schemaId, className, propertyName)
• Within this function, render the model into the "picking frame buffer"
• During this rendering, "pretend" the model had some sort of custom shader that just visualizes the metadata (sneaky...)
• Read the rendered metadata values from that "picking frame buffer"

Implementation draft

There is a Scene.pickMetadata function. This function receives the schemaId/className/propertyName of the metadata that is supposed to be picked.

Note: This is only a DRAFT. It assumes that the client already knows what should be picked. But the information about what can be picked may have to be exposed in one form or another. Maybe via some pickMetadataSchema function that returns the MetadataSchema of the picked object. Details TBD.

The function directly delegates to a Picking.pickMetadata function. This roughly follows the pattern of the other pick... functions: It sets up everything to prepare a "metadata picking pass", renders everything into a frame buffer, and returns the rendered pixels at the picked position.

Now, the crucial question is how the actual metadata property values end up in the frame buffer during this "metadata picking pass". And I know that the current solution is not right, but hope for feedback about better approaches. The current flow of events is:

• The Scene.pickMetadata function has to call model.resetDrawCommands(); to enforce re-building the model draw commands for with the "picking shader". It would be better to solve this with some model.drawCommands[42].pickMetadata = true;. But the structure of the model draw commands is... not obvious, and has many intricacies, and ... eventually, the shaders (or at least, parts of them) will have to be re-built, because the selection of the picked className/propertyName does happen at runtime after all.
• Therefore, the Picking.pickMetadata function puts the className/propertyName into the FrameState. This is probably far too global and too transient. But this information is passed in from the client, and has to end up in a shader. I don't know whether there is a better "path" to put this information into the right place. (There are some additional quirks there, related to OIT, but... let's ignore that for now)
• There is a new pipeline stage that is inserted into the ModelRuntimePrimitive - namely, the MetadataPickingPipelineStage. This will set the required #defines in the shader, based on the presence of the className/propertyName in the FrameState.
• The ModelFS.glsl now checks for the METADATA_PICKING define. When it is defined, then it skips certain operations, and instead, runs the metadataPickingStage. This obtains the "color" for the fragment from the metadata values (namely, the value that was selected via the METADATA_PICKING_PROPERTY_NAME define, that was set to be the propertyName). (Note: The fact that the ModelFS.glsl has to be broken up like that is not "nice", but this could and would be structured "more nicely", eventually)

Open quesitons

There are some obvious things that have to be handled. For example: How is the actual property value encoded into the RGBA components? Right now, it is just written into the R channel, just for a test. How could, for example, a DOUBLE metadata value be encoded? Do we need (or should be use) "floating point frame buffers"? And eventually, regardless of how the metadata value is encoded into the RGBA values, these values will have to be de-coded again in the Scene.pickMetadata function, to return the actual value to the user.

However, these are things that can be addressed after a general, sensible approach was developed. And my gut feeling is: The approach that is drafted here is likely not such a "general sensible" approach. I'm open for suggestions.

Example

The following is a Sandcastle that uses the Custom Shaders Property Textures Sandcatle data set to show the current state of the picking. It calls scene.pickMetadata using the class/property names of that data set, and prints the returned values to the console:

const viewer = new Cesium.Viewer("cesiumContainer", {
  globe: false
});
const scene = viewer.scene;

// MAXAR OWT Muscatatuk photogrammetry dataset with property textures
// containing horizontal and vertical uncertainty
const tileset = await Cesium.Cesium3DTileset.fromIonAssetId(2342602);
viewer.scene.primitives.add(tileset);
viewer.zoomTo(tileset);


const handler = new Cesium.ScreenSpaceEventHandler(scene.canvas);
handler.setInputAction(function (movement) {

  const classNameA = "r3dm_uncertainty_ce90sum";
  const propertyNameA = "r3dm_uncertainty_ce90sum";
  const pickedA = viewer.scene.pickMetadata(movement.endPosition, undefined, classNameA, propertyNameA);
  console.log("pickedA ", pickedA);

  const classNameB = "r3dm_uncertainty_le90sum";
  const propertyNameB = "r3dm_uncertainty_le90sum";
  const pickedB = viewer.scene.pickMetadata(movement.endPosition, undefined, classNameB, propertyNameB);
  console.log("pickedB ", pickedB);

}, Cesium.ScreenSpaceEventType.MOUSE_MOVE);

Again, these are just returned as 4-element byte arrays with the "value" encoded in the first byte, but details about the encoding/decoding can be decided later.

[github-actions]

Thank you for the pull request, @javagl!

✅ We can confirm we have a CLA on file for you.

[jjhembd]

I think the overall approach makes sense, although perhaps @lilleyse should chime in.

My main concern: calling model.resetDrawCommands from Scene.pickMetadata would effectively rebuild the shaders at every frame, which would be unworkably slow.

We only need to rebuild the shader when these values change:

  const schemaId = frameState.pickedMetadataSchemaId;
  const className = frameState.pickedMetadataClassName;
  const propertyName = frameState.pickedMetadataPropertyName;

Thinking about how an end user would pick: there would be some UI element letting them choose a class and property. These values would change only rarely--once every few seconds, at most. Then, once they mouse over the model, Scene.pickMetadata would be called every frame, but always with the same class and property values.

A faster approach might be:

1. Let the FrameState.pickedMetadata*** values persist (don't delete them in Picking.prototype.pickMetadata!).
2. Within Model.prototype.update, check for changes in the relevant FrameState values, and rebuild shaders only if needed. See the methods updateSceneMode, updateFog, updateVerticalExaggeration for an example.

Note: you will need to keep 2 compiled versions of the shader alive: one for the rendering pass, and one for the metadata picking pass. See FrameState.Passes. I haven't thought through the details yet, but I'm pretty sure you will need a new pickMetadata pass, similar to the pickVoxel pass.

The compiled shaders are stored on DrawCommands. See buildVoxelDrawCommands for how to set up a DrawCommand for each pass. It's actually relatively simple once you know the flow:

1. Clone the ShaderBuilder.
2. Add your PICKING define: shaderBuilderPick.addDefine("PICKING", ... ).
3. Compile it: const shaderProgramPick = shaderBuilderPick.buildProgram(context);
4. Clone the DrawCommand and attach the new shader.
5. Attach the DrawCommand to the primitive.

Then for each render pass, you need to select the appropriate DrawCommand based on the value of frameState.passes. See the end of VoxelPrimitive.update for an example.

[javagl]

I expected the resetDrawCommands to be one important point. It will "only" rebuild the shaders due to a pick, but when it is happening on a MOUSE_MOVE, then... yes, it would basically be "every frame".

Within Model.prototype.update, check for changes in the relevant FrameState values, and rebuild shaders only if needed. See the methods updateSceneMode, updateFog, updateVerticalExaggeration for an example.

I have seen the update... methods, but it seems like basically all of them are doing a resetDrawCommands when something changes. I'll probably have to zoom further into that, to understand how the recompilation for the state change can be avoided here.
Or more specifically: Where and how to keep both versions of the shaders in memory.

The answer may be "In a DrawCommand". But ... there are reasons why I did not try the DrawCommand-based approach. (Well, in fact, I did try it - this PR is only the condensed tip of an iceberg of internal work and experiments...).

One reason is that the code is ... let's call it intimidating. I roughly understand the idea behind DrawCommands. And I did start with some pickMetadata flag (similar to pickVoxel). But then I see code blocks like this, where some (undocumented) draw commands are juggled around, sometimes depending on pick, and sometimes depending on pickVoxel. How should pickMetadata be handled here? Probably, HDR does not matter for the picking, but what does "not matter" mean? (One could also think that OIT should not matter, but ... that turned out to be wrong...)

Another reason is that there apparently aren't really "draw commands" for the model - at least, not on the same level as for other elements: There's the ModelDrawCommand that ... manages stuff. It does not seem to be related to picking. But ... I don't know whether I'll have to sneak the "new" draw commands in there, and then suddenly have to add special handling for the 2D case, or know what the purpose of something like _skipLodStencilCommand is...

However, I'll try to read more, try to map your recommendations to the code, and see whether there is a way to achieve that pseudocode-ish model.drawCommands[42].pickMetadata = true; that may help to avoid the recomplication.

BTW: I've seen in your voxel picking PR that you started adding comments. That's good. When reading though long, undocumented functions, I usually cannot help myself, and try to make things easier to digest - even though this will likely be in vain...

[javagl]

Only a short update: Creating different draw commands for the picking/rendering case of a model is not as simple as it might be in other cases. As mentioned above: There aren't really "draw commands" for the model. There is the ModelDrawCommand that receives another draw command, and "derives" other draw commands from that (whatever that means), and occasionally pushes some commands into the commandList.

Trying to ignore the details (because there are waaaay too many, too undocumented details), one approach that I tried was to add a pickMetadata:boolean parameter to the buildDrawCommand function, and calling it once with false and once with true, to store the resulting draw command once as the runtimePrimitive.drawCommand and once as a runtimePrimitive.metadataPickingDrawCommand, and then call the pushCommands of either of them (depending on the frame state) to push the right commands to the commandList. That might be nearly working, but

1. I'm not sure whehther the ModelDrawCommand (or its drawCommand or any of its derived commands) require or allocate any kind of resources that would then be allocated twice. There's stuff like model._pipelineResources.push(vertexArray); in the buildDrawCommand function, but I'm not sure how deeply I should dive into that.
2. It is only nearly working. There are several places where other classes are doing runtimePrimitive.drawCommand.someProperty = someValue, which then of course is not applied to the metadataPickingDrawCommand.

Ignoring 1., I'll first try to "do whatever is done to the drawCommand also to the metadataPickingDrawCommand". That's ugly and increasing the complexity, but ... maybe it is what it is - decreasing the complexity could be the goal of dedicated refactoring PRs.

[javagl]

Another update. This can probably be ignored. It is rather a form of "Rubber Duck Debugging" right now.

---

The ModelSceneGraph is traversing the scene graph in buildDrawCommands to... build the draw commands. Hm. Makes sense.

For each ModelRuntimePrimitive that is found in a node, it creates PrimitiveRenderResources. (Does this cause something to be "allocated"? Who knows. All the properties of this structure are 'public', some of them are set directly, some of them are 'cloned', and some of them are set at other places of the code...)

The PrimitiveRenderResources contain a ShaderBuilder. The shaderBuilder of these render resources is used, modified, and extended in each "Pipeline Stage".

At some point, it does build a DrawCommand, using the buildDrawCommand function. Hm. Makes sense. This function receives the render resources, which contain the ModelRuntimePrimitive. The function passes references of elements of the render resources to other classes, and these references seem to be modified all over the place.
(It also modifies the render resources direcly - but only if (!is3D && !frameState.scene3DOnly && model._projectTo2D), which looks like if (is2D && isReally2D && isReallyHonestly2D) for me...)

---

tl;dr: The DrawCommand refers to ... "various stuff", that is modified "at various places". So...

• to have two commands (one for picking and one for rendering), it would be necessary to clone/copy the DrawCommand.
• to actually have different commands, it would be necessary to clone the ShaderBuilder, because this is modified everywhere.
• Since the ShaderBuilder is contained in the PrimitiveRenderResources, it would be necessary to clone/copy the PrimitiveRenderResources, because they are modified everywhere.
• Since the PrimitiveRenderResources are associated with the ModelRuntimePrimitive, it would be necessary to clone/copy that as well, ...
• ...
• Eventually, it would be necessary to copy/clone the whole scene graph...

Now, there's probably a point where cloning is not necessary. But given that undocumented modifications of undocumented properties are smeared all over the code, and nobody really knows what has to be cloned or not, the only reasonable approach for me right now is to try and clone things "until it works".

---

An anecdote: Many years ago, I created a small "renderer" library. And I did also use the concept of a (Draw)Command there. It was a tad simpler, though.

[javagl]

It also modifies the render resources direcly

I wrote this two days ago. And still, it took me several hours to find out why ~"sometimes, under certain conditions, the picking frame buffer was not filled with proper values". Right now, it seems like this is indeed related to the fact that the render resources - specifically, the primitiveRenderResources.boundingSphere - are modified when a draw command is created from them. Pain-in-the-back debugging...

      const bs = primitiveRenderResources.boundingSphere;
      const originalBoundingSphere = new BoundingSphere(bs.center, bs.radius);
      
      console.log("before building with pick=false ", primitiveRenderResources.boundingSphere.center);
      const drawCommand = ModelDrawCommands.buildModelDrawCommand(
        primitiveRenderResources,
        frameState,
        shaderBuilderRender,
        false
      );
      runtimePrimitive.drawCommand = drawCommand;
      console.log("after  building with pick=false ", primitiveRenderResources.boundingSphere.center);

      primitiveRenderResources.boundingSphere = originalBoundingSphere;
      const metadataPickingDrawCommand = ModelDrawCommands.buildModelDrawCommand(
        primitiveRenderResources,
        frameState,
        shaderBuilderPickMetadata,
        true
      );
      runtimePrimitive.metadataPickingDrawCommand = metadataPickingDrawCommand;
      console.log("after  building with pick=true  ", primitiveRenderResources.boundingSphere.center);

indicates that it "works" when saving and re-assigning that originalBoundingSphere, and it does not work when just creating the two draw commands in sequence.

Now...was that modification intentional? Does it serve a purpose? Should I avoid this modification? Does something else break when I prevent this modification? Can someone add a comment to the line if (!is3D && !frameState.scene3DOnly && model._projectTo2D) that clearly says
// When ... WHAT? ... then do ... WHAT?, because ... WHY?

This code is in dire need of cleanups. Having to hunt "bugs" that are caused by things like this will let any form of actual development come to a grinding halt.

[javagl]

However, here's a small update:

• The pickedMetadataSchemaId/ClassName/PropertyName are stored in the FrameState.
• The Model stores the same values, as _... private variables.
• The ModelRuntimePrimitive now stores two draw commands:
  • The drawCommand, which is the original ModelDrawCommand
  • A metadataPickingDrawCommand, which is the same thing, but where the shader was extended with the PICKING_METADATA information, including the frameState.pickedMetadataPropertyName to know what should be written into the frame buffer
• During the update of the model, it checks whether these values have changed. If they have changed, it does resetDrawCommands, triggering a rebuild, including a rebuild of the metadataPickingDrawCommand with the new pickedMetadataPropertyName

Technically, this seems to work in principle, with the caveats:

• The example sandcastle shown above contains lines like

  const pickedA = scene.pickMetadata(..., "propertyA");
  const pickedB = scene.pickMetadata(..., "propertyB");
  

  which means that it will (still) have to rebuild the draw commands on every mouse move. It needs the draw commands once with propertyA and once with propertyB.

• This will also (always) rebuild the main (non-picking) draw command. To minimize the amount of compilation, there should be two completely sepearate paths/states: For each

  • Model.resetDrawCommands()
  • Model_drawCommandsBuilt (boolean)
  • Model.buildDrawCommands(), ModelSceneGraph.buildDrawCommands()
  • ...

  there would have to be corresponding Model.resetMetadataPickingDrawCommands() etc. to just rebuild the picking ones, and not the rendering ones. (This means another increase of the complexity of the state space. So if this is done, I'll spam this with 40 lines of // Inlined comments explaining the purpose of all this...)

---

An aside: When picking the metadata values with debugShowBoundingVolume: true, it picks ... white pixels from the debug bounding volume 😬 This can probably be avoided easily. Just always set this flag to false in the metadataPickingDrawCommand. My concern is that there will be ~"other (similar) things" in that draw command that require special treatment (silhouettes, shadows, ...). I'll keep digging.

[javagl]

The last commit contains a draft for how the "encoding and decoding" of metadata values into the frame buffer could be handled. It is really only a draft, to quickly sync on the overall approach.

• The Scene.pickMetadata function now looks up the MetadataClassProperty, and stores it in the FrameState, like it already did with the schemaId/className/propertyName.
• Then the picking draw command is built, the shader is assembled based on the structure of the MetadataClassProperty.
  • The lines roughly have the pattern resultMetadataValues.x = (float)(metadatataPropertyValue.x) / 255.0 with the details depending on the MetadataClassProperty, i.e. its number of components of the property and whether it is normalized
• The actual value that is written into the frame buffer is always a vec4.
  • Right now, this will be converted to 32bit RGBA during rendering - we could consider floating-point frame buffers in the future

The values are still returned as a TypedArray (RGBA) right now - re-assembling/converting them back into the proper return type should be straightforward. The set of types that actually is supported for propety textures is very limited right now: Only SCALAR UINT8 (or up-to-4-element fixed length arrays thereof) are supported. This limitation should not "leak" through here, though. It should be prepared to convert "whatever it receives" into a VEC3 FLOAT32 and return it as a Cartesian3, for example. There probably already is some boilerplate code for stuff like this somewhere...

---

In the meantime, I'm less convinced of the overall approach than I was in the beginning. Funneling "arbitrary values" through the frame buffer, using the whole DrawCommand-infrastructure, feels like a "detour" (not to mention that it is a debugging nightmare). It would be great if there was an option to implement this more like pseudocode

void pick(Point p) {
    Ray r = scene.getRay(p);
    PickResult pickResult = scene.pick(ray);
    if (pickResult == null) return;
    Cartesian2 texCoords = computeTexCoordsAt(pickResult.triangle, pickResult.baryCoords);
    Texture texture = pickResult.model.getTexture();
    return texture.lookup(texCoords);
}

I even considered to just ditch the whole metadataPickingDrawCommand part, and actually do use a CustomShader for all this. Roughly as in

void pickMetadata(Point p) {
    const model = determineModel(p);
    CustomShader old = model.customShader;
    model.customShader = thatForMetadataPicking();
    render(model, pickFrameBuffer);
    model.customShader = old;
    ...
}

So basically really assign a CustomShader, temporarily, for the picking, and just use the existing infrastructure. The effect of the current metadataPickingDrawCommand is exactly the same: The model is rendered into the frame buffer as-if someone had assigned such a "metadata visualization CustomShader" - so why not just do this? I'll probably try it out, at least. It could be far simpler and less brittle than the current approach.

What's also concerning: There now are four properties in the FrameState that are related to metadata picking. The FrameState already is a pretty structure-free ~"collection of stuff that is needed somewhere", and I'd prefer to not put any additional properties in there.

[lilleyse]

Offline we talked about using the scene-level picking and derived command infrastructure instead of creating pick metadata commands in Model.

• Picking.prototype.pickMetadata would set frameState.passes.picking and frameState.passes.pickMetadata to true (frameState.passes.render would be false). See related discussion about pickVoxel in Add Scene._pickVoxelCoordinate to report voxel tile and sample numbers #11784 (comment).
• DrawCommand would have a new property pickMetadata. This would be a shader snippet like pickId.
• DerivedCommand would create derived commands for the pickMetadata pass.
• Not sure where schema id, class id, and property id would be set, either in Model when creating the pickMetadata shader string, or in DerivedCommand.

[javagl]

DerivedCommand would create derived commands for the pickMetadata pass.

The place where the current "derived commands" for picking are created is in the Scene.updateDerivedCommands function.

There are a few questions related to that. Note that I'm NOT asking these questions. Nobody has to even consider answering them. I'm only mentioning them, to make clear that I'm spending about 99% of my time with questions like these:

• The derived command is stored as derivedCommands.picking. (Well, not really: It is actually stored as derivedCommands.picking.pickCommand...) Who knows that the derivedCommands can contain a command called picking?
  • The documentation of the derivedCommands property is literally /** @private */ (!) - this, and only this. The presence of the picking is checked in executeCommand and executeIdCommand. And whether this picking is even considered there depends on whether there is a derivedCommands.logDepth or derivedCommands.hdr (both with zero documentation), and the exact configuration of frameState.useLogdepth, passes.pick, passes.pickVoxel, passes.pickDepth and scene._hdr. What does HDR have to do with that? Nothing. But everything that is happening there crucially depends on a deep understanding of how HDR is implemented - apparently.
• Where and when is updateDerivedCommands called?
  • I can find the respective places with a full-text search, and go up the call tree:

    View.insertIntoBin
        View.createPotentiallyVisibleSet
            Scene.executeWebVRCommands
                ...
            Scene.executeCommandsInViewport 
                ...
    

    The insertIntoBin could be called insertIntoBinAndCreateDerivedCommands, or better adhere to the Coding Guide which says "Functions should be cohesive; they should only do one task.". However, the last one is called in 10 different places. Now: Where and when is updateDerivedCommands called? What are the conditions for it being called? What is the effect of it being called? What exactly is "updated" there? Apparently, this is not just toggling some flag due to some state change. Instead, it seems to create the derived commands to begin with...
• What is the condition for that derivedCommands.picking to be created?
  • Yes, I see the condition - it is if (defined(command.pickId)). Now... who is assigning this pickId to the command at which point in time? And is this command there a command, or a derived command, or the command that is stored in the derived command of a command (!?!), or does this not matter because they all have the same pickId? Probably: The part that is relevant for the ~"model" is probably that of the PickingPipelineStage, where the pick ID is sneaked into the render resources, from where it is later read to be passed to the draw command that the model draw command and all its derived commands are created from.
• If I understood the intention here correctly, then there should be something that is "similar" to the pickId, insofar that it would be something like pickedMetadataInfo that is stored in a command and causes the creation of specific "derived commands" in the updateDerivedCommands function, and the execution of these derived commands in the executeCommand function. Some difficulties:
  • Regardless of what pickedMetadataInfo is, and how it arrives in one command: There will be one command that is executed and that needs this information. Maybe it has to be "forwarded" to other commands, like the pick IDs in the ClassificationModelDrawCommand.createPickCommands, where this ID is assigned to a pickColorCommand that is a clone of a derived command? All I can do is to shove that information into each shallowClone of a command, and cross my fingers.
  • Figuring out which command is executed is close to impossible. There is a function executeCommand(command, ... in Scene.js. But just because a function is called executeCommand and receives a command does not mean that this function executes the command. I'd encourage everybody to imagine what it would mean to write a short /** JSDoc comment **/ for that one...

[javagl]

@lilleyse

DerivedCommand would create derived commands for the pickMetadata pass.

Theoretically, this might be possible.

It is possible to create a createPickMetadataDerivedCommand, similar to the createPickDerivedCommand. And this involves a getPickMetadataShaderProgram, similar to the current getPickShaderProgram. But I won't push the state that I have locally and based on which I make this statement. (It involves a function sneakThatFunctionIntoTheFragmentShaderHopingThatSomeStringAppearsOnlyOnce...).

The problem is that at this point, only the shaderProgram is available (and specifically, no ShaderBuilder that could be used). I could now create a function like ShaderSoruce.insertMetadataPicking, and that would be pretty similar to ShaderSource.replaceMain. I could also think about solving this differently, by juggling the #defines of that ShaderProgram or whatnot.

But I'm pretty sure that I'm missing some point here, so I wanted to confirm whether "shader source code string manipulation with regex find-and-replace" is indeed the intended solution for this.

[javagl]

In the previous state, the approach was

• create two instances of the ModelDrawCommand
• in one instance, tweak the shader for metadata picking (while building it)
• store one as the standard runtimePrimitive.drawCommand, and one as the runtimePrimitive.metadataPickingDrawCommand

This duplication was probably "too high", so to speak, and did lead to some issues. It's hard to tell for sure what exactly was duplicated there to begin with. And it's hard to say where the frameState.passes.pick flag had to be taken into account. Apparently, commands had been derived from the command with the modified shader code, leading to ... effects that could nearly be "expected" ... like ... the metadata values depending on the time of the day, due to lighting changes and such 😬

---

The last commit changes this, based on the recommendation by @lilleyse , to do the metadata handling in derived commands.

^{The current state contains several 'markers', specifically, XXX_METADATA_PICKING, where documentation has to be added or debug logs have to be removed - ignore that for now...}

There now is a MetadataPickingPipelineStage that just inserts the code for the metadata picking into the model shader. Initially and effectively, this code is this:

void metadataPickingStage(Metadata metadata, MetadataClass metadataClass, inout vec4 metadataValues) {
    float value = 0.0;
    metadataValues.x = 0.0;
    metadataValues.y = 0.0;
    metadataValues.z = 0.0;
    metadataValues.w = 0.0;
}

but all the parts that are varying are represented as #defines.

In the Scene.updateDerivedCommands function, a new command is created - specifically, the derivedCommands.pickingMetadata command (that is later executed in executeCommand during metadata picking).

That derived command is created by DerivedCommands.createPickMetadataDerivedCommand, which creates a new command where the shader code (shown above) is filled with life, by setting the proper #defines. The main part of that functionality is implemented here, which basically looks at the MetadataClassProperty that defines the picked metadata property, and sets all the #defines that cause that metadataPickingStage function to eventually look, for example, like this:

void metadataPickingStage(Metadata metadata, MetadataClass metadataClass, inout vec4 metadataValues) {
    ivec2 value = ivec2(metadata.exampleProperty);
    metadataValues.x = (float)(value.x / 255.0);
    metadataValues.y = (float)(value.y / 255.0);
    metadataValues.z = 0.0;
    metadataValues.w = 0.0;
}

These values end up in the frame buffer, from where they are read, and translated back into the actual metadata value (a 2-element array in the above example).

Now. Funneling that data through the frame buffer seems quirky. Assembling the proper shader for that based on #defines even more so. And I still have to spend more time to better understand what's actually going on with these 'derived commands'.
But I'll leave it at that for now, just to ask:

Is this the intended approach?

---

EDIT, a small note: Right now, this is still rebuilding the draw commands when the picked metadata property changes. Updating the "minimal set" (of derived commands?) upon changes still remains to be implemented...

[javagl]

The last point of rebuilding the draw commands was now addressed as part of a minor ... "consolidation".

From my point of view, the functionality that is added with this PR does not warrant the number and complexity of the changes that have been necessary to accomplish it. But doing what is necessary to decrease the complexity cannot be part of this PR. Maybe I'll add follow-up PRs similar to #12058 , but it's hard to make promises here.

[ggetz]

Thanks @javagl, I did a final once over this PR to ensure this is good to merge for tomorrow's release.

• I've updated CHANGES.md to document the changes to the public API, based on my understanding of the updates. Please make sure to include updates to CHANGES.md in the future, even for features marked as experimental, as it acts as a historical record.
• I've gone and verified all existing picking examples, and everything seems to be working as before.
• I also tested out the new functionality with the Sandcastle below. I noticed that there was a big hit to the frame rate– When mousing around the scene, I'm getting 0-1 FPS. Is this expected?
• I also made several review comments. If we need to get this in for tomorrow's release, we can move forward without them (and therefore approved this PR), but I would like to see a cleanup PR follow-up soon after the release.

---

Sandcastle example using existing sample data:

const viewer = new Cesium.Viewer("cesiumContainer", {
  terrain: Cesium.Terrain.fromWorldTerrain(),
});
const scene = viewer.scene;
scene.globe.depthTestAgainstTerrain = false;

scene.debugShowFramesPerSecond = true;

let tileset;
try {
  // MAXAR OWT Muscatatuk photogrammetry dataset with property textures
  // containing horizontal and vertical uncertainty
  tileset = await Cesium.Cesium3DTileset.fromIonAssetId(2342602);
  viewer.scene.primitives.add(tileset);
  viewer.zoomTo(tileset);
} catch (error) {
  console.log(`Error loading tileset: ${error}`);
}

const shaders = {
  NO_TEXTURE: undefined,
  UNCERTAINTY_CE90: new Cesium.CustomShader({
    fragmentShaderText: `
void fragmentMain(FragmentInput fsInput, inout czm_modelMaterial material)
{
  int horizontalUncertainty = fsInput.metadata.r3dm_uncertainty_ce90sum;
  material.diffuse = vec3(float(horizontalUncertainty) / 255.0);
}
      `,
  }),
  UNCERTAINTY_LE90: new Cesium.CustomShader({
    fragmentShaderText: `
void fragmentMain(FragmentInput fsInput, inout czm_modelMaterial material)
{
  int verticalUncertainty = fsInput.metadata.r3dm_uncertainty_le90sum;
  material.diffuse = vec3(float(verticalUncertainty) / 255.0);
}
      `,
  }),
  // combined uncertainty
  UNCERTAINTY: new Cesium.CustomShader({
    fragmentShaderText: `
void fragmentMain(FragmentInput fsInput, inout czm_modelMaterial material)
{
  int uncertainty = fsInput.metadata.r3dm_uncertainty_ce90sum + fsInput.metadata.r3dm_uncertainty_le90sum;
  material.diffuse = vec3(float(uncertainty) / 255.0);
}
      `,
  }),
};

Sandcastle.addDefaultToolbarMenu([
  {
    text: "Horizontal Uncertainty",
    onselect: function () {
      tileset.customShader = shaders.UNCERTAINTY_CE90;
    },
  },
  {
    text: "Vertical Uncertainty",
    onselect: function () {
      tileset.customShader = shaders.UNCERTAINTY_LE90;
    },
  },
  {
    text: "Combined Uncertainty",
    onselect: function () {
      tileset.customShader = shaders.UNCERTAINTY;
    },
  },
  {
    text: "No Uncertainty",
    onselect: function () {
      tileset.customShader = shaders.NO_TEXTURE;
    },
  },
]);
tileset.customShader = shaders.UNCERTAINTY_CE90;

// Create an HTML element that will serve as the
// tooltip that displays the metadata information
function createTooltip() {
  const tooltip = document.createElement("div");
  viewer.container.appendChild(tooltip);
  tooltip.style.backgroundColor = "black";
  tooltip.style.position = "absolute";
  tooltip.style.left = "0";
  tooltip.style.top = "0";
  tooltip.style.padding = "14px";
  tooltip.style["pointer-events"] = "none";
  tooltip.style["block-size"] = "fit-content";
  return tooltip;
}
const tooltip = createTooltip();

// Show the given HTML content in the tooltip
// at the given screen position
function showTooltip(screenX, screenY, htmlContent) {
  tooltip.style.display = "block";
  tooltip.style.left = `${screenX}px`;
  tooltip.style.top = `${screenY}px`;
  tooltip.innerHTML = htmlContent;
}

// Create an HTML string that contains information
// about the given metadata, under the given title
function createMetadataHtml(title, metadata) {
  if (!Cesium.defined(metadata)) {
    return `(No ${title})<br>`;
  }
  const propertyKeys = metadata.getPropertyIds();
  if (!Cesium.defined(propertyKeys)) {
    return `(No properties for ${title})<br>`;
  }
  let html = `<b>${title}:</b><br>`;
  for (let i = 0; i < propertyKeys.length; i++) {
    const propertyKey = propertyKeys[i];
    const propertyValue = metadata.getProperty(propertyKey);
    html += `&nbsp;&nbsp;${propertyKey} : ${propertyValue}<br>`;
  }
  return html;
}


const handler = new Cesium.ScreenSpaceEventHandler(viewer.scene.canvas);
handler.setInputAction(function (movement) {

  const classNameA = "r3dm_uncertainty_ce90sum";
  const propertyNameA = "r3dm_uncertainty_ce90sum";
  const classNameB = "r3dm_uncertainty_le90sum";
  const propertyNameB = "r3dm_uncertainty_le90sum";
  const pickedA = viewer.scene.pickMetadata(
    movement.endPosition,
    undefined,
    classNameA,
    propertyNameA
  );
  const pickedB = viewer.scene.pickMetadata(
    movement.endPosition,
    undefined,
    classNameB,
    propertyNameB
  );

  let tooltipText = "";
  tooltipText += propertyNameA + ": " + pickedA + "<br>";
  tooltipText += propertyNameB + ": " + pickedB + "<br>";

  const screenX = movement.endPosition.x;
  const screenY = movement.endPosition.y;
  showTooltip(screenX, screenY, tooltipText);
}, Cesium.ScreenSpaceEventType.MOUSE_MOVE);

[ggetz]

@private means the member will not be exposed in the API documentation, and can change without warning or deprecation. It does not mean it can't be used within other parts of CesiumJS.

Assuming that is how this function is intended to be used, I think this line is redundant and can be removed.

[ggetz]

Additionally, assuming these utility functions are generalizable and applicable to MetadataType, should they live in MetadataType instead? For example convertToObjectType seems quite general to dealing with metadata.

[javagl]

I tried to write them in a somewhat general form. And there already is quite a lot of existing functionality - for example, in https://github.com/CesiumGS/cesium/blob/e3e98371830a85d653c2f9e1f34f382c84984bce/packages/engine/Source/Scene/MetadataTableProperty.js . In both cases, ("typed") metadata values have to be read from binary data.

There are some subtle differences, though. Pulling out the parts that are really generic (and using these parts in both the MetdataPicking and the MetadataTableProperty) could be part of a follow-up/cleanup PR.

[javagl]

This is now summarized in #12225

[ggetz]

Remove TODO comments like this from the code.

[javagl]

Done - now throws a RuntimeError

[ggetz]

Should we be using MetadataType rather than hardcoding strings? Likewise, should we enforce that by marking the type of type as MetadataType?

[javagl]

Done

[ggetz]

Spec file organization should mirror the Source directory organization.

The structure of these specs allude to a file which would exist at packages/engine/Source/Scene/Model/pickMetadata.js, which doesn't exist.

[javagl]

This was mimicking the existing pickModelSpec.js. I desired, I can merge the specs into the SceneSpec, but ... that will be >1000 lines to a spec with already >2300 lines....

[ggetz]

Yes, but pickModelSpec.js is a standalone file.

Let's go with what is described in the guide and move the code. I agree it is a large file, but to break from recommendations, we should propose an update to the guide separate from this PR, which is already on the larger side.

[javagl]

OK, just moved it over into SceneSpec and removed pickMetadataSpec.js

[javagl]

I addressed most of the inlined comments.

I also tested out the new functionality with the Sandcastle below. I noticed that there was a big hit to the frame rate– When mousing around the scene, I'm getting 0-1 FPS. Is this expected?

No. From the description, it sounds like there might be an issue with the shader caching: Some of the earlier discussion revolved around the point of avoiding unnecessary recompilations, exactly for that case: The shaders that are built for picking the respective metadata property are supposed to be only built once and then re-used.

There had been a last-minute change in that area, based on another comment - maybe something there had an unintended side-effect. But I could not reproduce the frame rate issue locally until now: With the sandcastle that you posted, and constant mouse movement, I got this:

with stable 59-60 FPS.

I'll take a note in a follow-up issue to investigate/check whether something went wrong with the shader caching.
This will also include notes about possible cleanups for the Spec structure, and possible code-reuse in the MetadataPicking class.

[javagl]

The build just failed with an error, saying that a required parameter cannot follow an optional one.

This was caused by the change

[rem:] * @param {string|undefined} schemaId The ID of the metadata schema to pick values
[add:] * @param {string} [schemaId] The ID of the metadata schema to pick values

suggested in #12075 (comment) . This is related to the point that was already brought up in #12075 (comment) .

The [angleBrackets] explicitly denote an optional parameter. But the schemaId in this case is not optional - it is required, but can still be undefined, which here serves the purpose of being a "wildcard" (i.e. ignoring the schema.id).

I'll also add that to the notes for the follow-up PR.

[ggetz]

I'll take a note in a follow-up issue to investigate/check whether something went wrong with the shader caching.

Thanks! Since this affects behavior, I want to be sure that this is documented if not addressed immediately.

The [angleBrackets] explicitly denote an optional parameter. But the schemaId in this case is not optional - it is required, but can still be undefined, which here serves the purpose of being a "wildcard" (i.e. ignoring the schema.id).

OK, makes sense 👍

[javagl]

I summarized the open points here in a follow-up issue at #12225 (most of them refer to this PR, but for some of them, the specifics of the connection to the GPM support are relevant)

[javagl]

@ggetz I think that the inlined comments have been addressed (or moved into a follow-up issue, accordingly).

I had to re-run CI a few times - there seems to be s somewhat flaky-shaky test. But that seems to be independent of this PR.

[ggetz]

Thanks @javagl! Yes, the flakey should not be due to this PR.

[javagl]

I also tested out the new functionality with the Sandcastle below. I noticed that there was a big hit to the frame rate– When mousing around the scene, I'm getting 0-1 FPS. Is this expected?

I could now reproduce this. By hitting F12 😬

So... when the DevTools are closed, I receive a smooth framerate of ~59, regardless of the mouse movement.
When the DevTools are open, it goes down to ~1 FPS during continuous mouse movement.

I'll still check whether there's something wrong with the shader caching. And I know that the DevTools do not come "for free". But that drop is pretty significant...

[javagl]

Now I'm at the point where I think that the Chrome DevTools are just fooling me. When running with the DevTools open, it drops to ~1FPS. But when also running the profiler, I get a stable 59FPS again 🤡

However, I did check two things:

• It does not re-compile the shaders. So it is re-using the shaders for the metadata picking, caching them based on the schemaId-className-propertyName key
• Even when just calling scene.pick and showing the resulting object in a tooltip text, I receive a noticable FPS drop (to ~35-40). (Note that this also only happens when the DevTools are open)

I'd lean towards considering this as a DevTools glitch, somewhat unrelated to the pickMetadata function, unless someone thinks that this may warrant further investigation.

[ggetz]

I could now reproduce this. By hitting F12

I likely had the developer tools open 😅

I'd lean towards considering this as a DevTools glitch

Non-ideal to be sure, but this is not the first time we've seen happen. I believe @jjhembd ran into a similar issue when profiling 3D Tiles load times.