Feat: Add distort shader #271

lewispeel · 2024-05-09T12:53:23Z

I recently created a shader In Lightning 2 to distort a texture inside itself and ported it to Lightning 3. It allows you to define new coordinates for top-left, top-right, bottom-right and bottom-left inside it's original dimensions.

Changes

Added DistortShader to WebGL shaders (based on https://www.shadertoy.com/view/lsVSWW).
Added Point interface to CommonTypes to make shader params typesafe.
Added an example in examples/tests/distort-shader that shows using pixel values and normalized values.
Updated CoreShaderManager to include DistortShader as an option.

How to test

Run pnpm start
In the browser, navigate to http://localhost:5173/?test=distort-shader

Usage

You can use explicit pixel values or normalized values e.g. { x: 0.5 } will be 50% of the texture width. To use normalized values, add normalized: true to the shader parameters.

const size = 500;

const parent = renderer.createNode({
  parent: testRoot,
});

// distorted using pixel values
const pixels = renderer.createNode({
  x: 100,
  y: 100,
  width: size,
  height: size,
  parent,
  src: `https://picsum.photos/${size}/${size}`,
  shader: renderer.createShader('DistortShader', {
    topLeft: { x: 20, y: 50 },
    topRight: { x: 400, y: 0 },
    bottomRight: { x: 350, y: 300 },
    bottomLeft: { x: 50, y: 350 },
  }),
});

// distorted using normalized values
const normalized = renderer.createNode({
  x: 700,
  y: 100,
  width: size,
  height: size,
  parent,
  src: `https://picsum.photos/${size}/${size}`,
  shader: renderer.createShader('DistortShader', {
    normalized: true,
    topLeft: { x: 0.1, y: 0.1 },
    topRight: { x: 0.75, y: 0.2 },
    bottomRight: { x: 0.95, y: 0.7 },
    bottomLeft: { x: 0.05, y: 0.9 },
  }),
});

Output

Examples using pixel values (left), normalized values (right). The dark rectangle are to illustrate where the original dimensions of the image textures.

src/core/renderers/webgl/shaders/Distort.ts

examples/tests/distort.ts

src/core/renderers/webgl/shaders/Distort.ts

src/render-drivers/utils.ts

frank-weindel · 2024-05-09T19:57:12Z

src/core/renderers/webgl/shaders/Distort.ts

+      topLeft: props.topLeft || { x: 0, y: 0 },
+      topRight: props.topRight || { x: 1920, y: 0 },
+      bottomRight: props.bottomRight || { x: 1920, y: 1080 },
+      bottomLeft: props.bottomLeft || { x: 0, y: 1080 },


These defaults are a bit too specific. I think one thing you might be able to do to achieve this dynamically is to allow null as value for each of these keys and make null the default. Then in bindProps I think you could check for null and dynamically create the defaults using the $dimensions property which at that point is reliable. (Going to leave a separate comment in regards to your request for comments on potentially using normalized 0.0 -> 1.0 values)

Done. I think that approach is nice. I was calculating the normalized values in the shader code but this works. To get this working using null I had to modify the Point class to accept number and null types. Allowing null feels a bit weird to me. I could keep it as number only and the default be zero, then if I see a zero size width or height value assume its been left as zero on purpose and use the full width/height? What do you think?

I'd regard the whole Point as a unit. The type of each corner in that cause would be Point | null. So if you need to provide an x, you also need to provide the y.

Ah that makes sense, I'll change it 👍

@frank-weindel Done!

frank-weindel · 2024-05-09T21:25:05Z

@lewispeel Awesome work. Thanks for the contribution. Regarding your question about potentially using relative normalized values instead of absolute position values, I'd say allow both. I can see both methods coming in handy depending on the situation. You could base it off of a new boolean prop flag (normalized perhaps) and do all the necessary resolutions in bindProps().

Please let me know if you need assistance generating the certified snapshot needed for this. You need to run the visual regression test runner in --ci --capture mode.

revert formatting

lewispeel · 2024-05-10T12:14:29Z

@frank-weindel Thanks for the feedback! I've got it working with both pixel and normalized values. Something to note, see line 132 in DistortShader - there's a bug when the values create a rectangle.. not sure why this is but I've added a workaround and would appreciate some feedback.

erikhaandrikman · 2024-05-10T12:33:28Z

src/core/renderers/webgl/shaders/DistortShader.ts

+    // if the dimensions create a rectangle, nothing is rendered
+    // adding a small amount to one of the x values resolves this
+    if (topLeft[0] === bottomLeft[0] && topRight[0] === bottomRight[0]) {
+      topLeft[0] = +0.0001;


Small typo here, you're now assigning 0.0001 to top-left x position. You probably meant:
topLeft[0] += 0.0001

Good catch, thanks @erikhaandrikman - I removed this block as the fix you suggested below means I don't need this anymore.

erikhaandrikman · 2024-05-10T13:47:56Z

src/core/renderers/webgl/shaders/DistortShader.ts

+        return a.x * b.y - a.y * b.x;
+      }
+
+      vec2 invBilinear(in vec2 p, in vec2 a, in vec2 b, in vec2 c, in vec2 d ){


To prevent the rendering issues when the axis are perpendicular ( like you mentioned in your comment ) you can replace in inverse bilinear function for:

vec2 invBilinear( in vec2 p, in vec2 a, in vec2 b, in vec2 c, in vec2 d ) { vec2 e = b-a; vec2 f = d-a; vec2 g = a-b+c-d; vec2 h = p-a; float k2 = xross( g, f ); float k1 = xross( e, f ) + xross( h, g ); float k0 = xross( h, e ); float w = k1*k1 - 4.0*k0*k2; if( w<0.0 ) return vec2(-1.0); w = sqrt( w ); // will fail for k0=0, which is only on the ba edge float v = 2.0*k0/(-k1 - w); if( v<0.0 || v>1.0 ) v = 2.0*k0/(-k1 + w); float u = (h.x - f.x*v)/(e.x + g.x*v); if( u<0.0 || u>1.0 || v<0.0 || v>1.0 ) return vec2(-1.0); return vec2( u, v ); }

Amazing! Thanks @erikhaandrikman - this is fixed.

frank-weindel · 2024-05-13T13:40:05Z

@lewispeel This needs a snapshot generated with:

pnpm test:visual --ci --capture

Check out the DOCKER.md file for more information on how to set things up for it.

lewispeel · 2024-05-13T14:42:10Z

@erikhaandrikman I've just spotted a problem.. this demo works fine on single nodes with no children, but if I create a node with child nodes (requiring rtt=true), the texture coords are flipped vertically... topLeft becomes bottomLeft topRight becomes bottomRight. Do you know any reason why this might be?

To test this, comment out lines 93-95 in tests/distort-shader.ts - it should apply the effect to only the top left but applies it to the bottom left. If you then comment out rtt:true on line 89 and 99-133 and the effect works again.

lewispeel · 2024-05-13T14:59:09Z

@lewispeel This needs a snapshot generated with:
pnpm test:visual --ci --capture
Check out the DOCKER.md file for more information on how to set things up for it.

Thanks Frank, I'll wait for Erik to feed back on the issue above and I'll make sure to run the visual tests 👍

erikhaandrikman · 2024-05-14T15:26:08Z

@erikhaandrikman I've just spotted a problem.. this demo works fine on single nodes with no children, but if I create a node with child nodes (requiring rtt=true), the texture coords are flipped vertically... topLeft becomes bottomLeft topRight becomes bottomRight. Do you know any reason why this might be?

To test this, comment out lines 93-95 in tests/distort-shader.ts - it should apply the effect to only the top left but applies it to the bottom left. If you then comment out rtt:true on line 89 and 99-133 and the effect works again.

probably due to the fact that the render texture is render flipped ( and coords being restored before pushing them to the buffer )

renderer/src/core/renderers/webgl/WebGlCoreRenderer.ts

Line 306 in 70af9d4

[texCoordY1, texCoordY2] = [texCoordY2, texCoordY1];

i can look into this

erikhaandrikman · 2024-06-26T09:00:55Z

We're encountering some projection issues, primarily with text, when attempting to flip the rendering in render texture nodes. As a temporary solution, we've been rendering the contents to an extra childNode (with RTT enabled) and flipping the rendering on the Y-axis by setting scaleY: -1

const distorted = renderer.createNode({
    x: 700,
    y: 100,
    width: size,
    height: size,
    parent,
    src: `https://picsum.photos/${size}/${size}`,
    shader: renderer.createShader('DistortShader', {
      normalized: true,
      topLeft: { x: 0.1, y: 0.1 },
      topRight: { x: 0.75, y: 0.2 },
      bottomRight: { x: 0.95, y: 0.7 },
      bottomLeft: { x: 0.05, y: 0.9 },
    }),
  });

  // distorted children inside a distorted container (requires rtt=true)
  const nested = renderer.createNode({
    x: 1300,
    y: 100,
    width: size,
    height: size,
    parent,
    rtt: true,
  });

  const nestedInner = renderer.createNode({
    width: size,
    height: size,
    parent: nested,
    rtt: true,
    color:0xffffffff,
    shader: renderer.createShader('DistortShader', {
      normalized: true,
      topLeft: { x: 0.1, y: 0.1 },
      topRight: { x: 0.75, y: 0.2 },
      bottomRight: { x: 0.95, y: 0.7 },
      bottomLeft: { x: 0.05, y: 0.9 },
    }),
    scaleY: -1,
  });

  const nestedBg = renderer.createNode({
    width: size,
    height: size,
    color: 0x990000ff,
    parent: nestedInner
  });

  const nestedHolder = renderer.createNode({
    width: size,
    height: size,
    parent: nestedInner,
    scaleY:-1,
    color: 0x99000000
  });
  
  // a distorted image inside a distorted container
  const image = renderer.createNode({
    x: size / 2,
    y: size / 2,
    width: size / 2,
    height: size / 2,
    mount: 0.5,
    parent: nestedHolder,
    src: `https://picsum.photos/${size}/${size}`,
    shader: renderer.createShader('DistortShader', {
      normalized: true,
      topLeft: { x: 0.1, y: 0.1 },
      topRight: { x: 0.75, y: 0.2 },
      bottomRight: { x: 0.95, y: 0.7 },
      bottomLeft: { x: 0.05, y: 0.9 },
    }),
  });

wouterlucas · 2024-07-25T14:35:38Z

@erikhaandrikman can we move this along with known limitations of RTT?

lewispeel added 5 commits May 9, 2024 13:42

add distort shader and example

1861614

change from arrays to objects

281957e

add defaults to docs

889c16c

add link to shadertoy demo this was based on

e87b707

update example comments

58a9c2c