Skip to content

Commit

Permalink
Document conversion pipeline (#3719)
Browse files Browse the repository at this point in the history 
* document conversion pipeline

* fix doc build

* fix block

---------

Co-authored-by: SimonDanisch <[email protected]>
  • Loading branch information
@ffreyer @SimonDanisch
ffreyer and SimonDanisch authored Mar 20, 2024
1 parent 2176f0f commit 67ebcf2
Showing 1 changed file with 249 additions and 0 deletions.
249 changes: 249 additions & 0 deletions docs/explanations/conversion_pipeline.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,249 @@
# Conversion Pipeline and Coordinate spaces

## Conversion Pipeline

The following graphic sketches out the conversion pipeline of data given to a plot with `space = :data` for GL backends.

\begin{examplefigure}{}
```julia
#hideall
using GLMakie, LinearAlgebra

function myarrows!(scene, ps; kwargs...)
ends = map(ps) do ps
output = Int[]
for i in eachindex(ps)
isnan(ps[i]) && push!(output, i-1)
end
push!(output, length(ps))
output
end

dict = Dict(kwargs)
endpoints = map((ps, is) -> ps[is], ps, ends)
dirs = map(
(ps, is) -> -Makie.quaternion_to_2d_angle.(Makie.to_rotation(normalize.(ps[is] .- ps[is .- 1]))),
ps, ends)
cols = map(is -> dict[:color] isa Vector ? dict[:color][is] : dict[:color], ends)

lines!(scene, ps; kwargs...)
scatter!(
scene, endpoints, marker = Makie.BezierUTriangle, color = cols,
rotations = dirs
)
end

scene = Scene(size = (1220, 200))
campixel!(scene)

# init space label data
spacing = 80
y0 = 80
ps = Observable(Point2f.(1:8, y0))
# spaces = [":data", "transformed64", "world", "eye", ":clip", "screen"]
spaces = ["plot.args", "plot.converted", "transformed64", "transformed32", "world", "eye", "clip", "screen"]

# plot space labels
p = text!(
scene, ps, text = spaces, align = (:left, :center),
fontsize = 20,
color = [:black, :black, :gray, :gray, :gray, :gray, :gray, :gray]
)

# update space label data & derive arrows
centers = Observable(Point2f[])
text_centers = Observable(Point2f[])
map!(ps, p.plots[1][1]) do gcs
edge = -spacing + 10
xvals = Float64[]
xcenters = Float64[]

for gc in gcs
bb = Makie.text_boundingbox(gc, Quaternionf(0,0,0,1))
left = spacing + edge - minimum(bb)[1]
push!(xvals, left)
push!(xcenters, left + 0.5 * widths(bb)[1])
edge = left + widths(bb)[1]
end

text_centers[] = Point2f.(xcenters, y0)
centers[] = Point2f.(xvals[2:end] .- 0.5spacing, y0)

return Point2f.(xvals, y0)
end

arrowpos = map(centers) do centers
half = 0.5 * spacing - 5
ps = Point2f[]
lws = Float64[]
for center in centers
x, y = center
push!(ps, Point2f(x-half, y), Point2f(x+half-5, y), Point2f(NaN))
# push!(ps, Point2f(x-half, y), Point2f(x+half-8, y))
# push!(ps, Point2f(x+half-10, y), Point2f(x+half, y))
# push!(lws, 2, 2, 12, 0)
end
ps
end

# plot arrows
myarrows!(
scene, arrowpos, linewidth = 2,
color = [c for c in [:red, :red, :red, :green, :green, :green, :gray] for _ in 1:3]
)

# transformation labels
transformations = ["convert_arguments", "transform_func", "Float32Convert", "model", "view", "projection", "viewport"]
trans_offset = 25
text!(
scene, centers, text = transformations, align = (:center, :center),
offset = (0, trans_offset),
color = [:black, :orange, :black, :orange, :cyan, :cyan, :gray]
)

# Transformation
bracket_offset = Point2f(0, trans_offset + 15)
trans_bracket_pos = map(centers) do cs
[cs[2] + bracket_offset, cs[4] + bracket_offset]
end
text!(
scene, trans_bracket_pos, text = ["Transformation" for _ in 1:2],
color = :orange, align = (:center, :bottom))

# Camera
cam_bracket_pos = map(centers) do cs
(cs[5] + bracket_offset, cs[6] + bracket_offset)
end
bracket!(
scene,
cam_bracket_pos,
text = "Camera",
color = :cyan, textcolor = :cyan
)

# CPU
dx = 10; dy = -20
cpu_bracket_pos = map(text_centers) do ps
(ps[1] .+ (dx, dy), ps[4] .+ (-dx, dy))
end
bracket!(
scene,
cpu_bracket_pos,
text = "CPU",
color = :red, textcolor = :red,
orientation = :down
)

# GPU
gpu_bracket_pos = map(text_centers) do ps
(ps[4] .+ (dx, dy), ps[7] .+ (-dx, dy))
end
bracket!(
scene,
gpu_bracket_pos,
text = "GPU",
color = :green, textcolor = :green,
orientation = :down
)

# Internal
internal_bracket_pos = map(text_centers) do ps
(ps[7] .+ (dx, dy), ps[8] .+ (-dx, dy))
end
bracket!(
scene,
internal_bracket_pos,
text = "GPU Internal",
color = :gray, textcolor = :gray,
orientation = :down
)

# Float32Convert
f32_ps = map(text_centers) do cs
x1, y1 = cs[2]
x5, y5 = cs[3]
x2, y2 = 0.5 * (cs[3] .+ cs[4])
x3, y3 = cs[6]
x4, y4 = 0.5 * (cs[4] .+ cs[5])
Point2f[
(x1, y1+15), (x1, y1+65), (NaN, NaN),
(x1+45, y1+83), (x5, y5+83), (x5, y2+25), (x5+45, y2+25), (NaN, NaN),
(x5-50, y2+25), (x5-10, y2+25), (NaN, NaN),
(x2+20, y2+40), (x2+20, y2+100), (x3, y3+100), (x3, y3+80), (NaN, NaN),
(x2+60, y2+25), (x4-15, y4+25)
]
end
myarrows!(scene, f32_ps, color = :gray)
text!(
scene, map(ps -> ps[2], f32_ps), text = "ax.finallimits",
align = (:center, :bottom), offset = Vec2f(0, 10)
)

scene
```
\end{examplefigure}

### Argument Conversions

When calling a plot function, e.g. `scatter!(axis_or_scene, args...)` a new plot object is constructed.
The plot object keeps track of the input arguments in `plot.args`, promoting them to observables if need be.
It also keeps track of a type normalized set of inputs in `plot.converted` which are generated from `plot.args` using various `convert_arguments()` functions.
Generally speaking these functions either dispatch on the plot type or the result of `conversion_trait(PlotType, args...)`, i.e. `convert_arguments(type_or_traint, args...)`.
They are expected to generalize and simplify the structure of data given to a plot while leaving the numeric type as either a Float32 or Float64 as appropriate.

### Transformation Objects

The remaining transformed versions of data are not accessible, but rather abstract representations which the data goes through.
As such they are named based on the coordinate space they are in and grayed out.
Note as well that these representations are only relevant to primitive plots like lines or mesh.
Ignoring `Float32Convert` for now, the next two transformations are summarized under the `Transformation` object present in `plot.transformation` and `scene.transformation`.

The first transformation is `transformation.transform_func`, which holds a function which is applied to a `Vector{Point{N, T}}` element by element.
It is meant to resolve transformations that cannot be represented as a matrix operations, for example moving data into a logarithmic space or into Polar coordinates.
They are implemented using the `apply_transform(func, data)` methods.
Generally we also expect transform function to be (partially) invertable and their inverse to be returned by `inverse_transform(func)`.

The second transformation is `transformation.model`, which combines `translate!(plot, ...)`, `scale!(plot, ...)` and `rotate!(plot, ...)` into a matrix.
The order of operations here is fixed - rotations apply first, then scaling and finally translations.
As a matrix operation they can and are handled on the GPU.

### Float32Convert

Nested between `transform_func` and `model` is the application of `scene.float32convert`.
Its job is to bring the transformed data into a range acceptable for `Float32`, which is used on the GPU.

Currently only `Axis` actually defines this transformation.
When calling `plot!(axis, ...)` it takes a snapshot of the limits of the plot using `data_limits(plot)` and updates its internal limits.
These are combined with other sources to generate `axis.finallimits`.
When setting the camera matrices `axis.finallimits` gets transformed by `transform_func` and processed by `scene.float32convert` to generate a valid Float32 range for the camera.
This processing will update the `Float32Convert` if needed.

With respect to the conversion pipeline the `Float32Convert` is a linear function applied to transformed data using `f32_convert(scene, data)`.
After the transformation, data strictly uses Float32 as a numeric type.

Note that since the `Float32Convert` is based on and transforms the limits used to create the camera (matrices), it should technically act between `model` and `view`.
In fact, this order is used for CairoMakie and some CPU projection code.
For the GPU however, we want to avoid applying `model` on the CPU.
To do that we calculate a new model matrix using `new_model = patch_model(scene, model)`, which acts after `Float32Convert`.

### Camera

Next in our conversion pipeline are the camera matrices tracked in `scene.camera`.
Their job is to transform plot data to a normalized "clip" space.
While not consistently followed, the `view` matrix is supposed to adjust the coordinate system to that of the viewer and the `projection` matrix is supposed to apply scaling and perspective projection if applicable.
The viewers position and orientation is set by either the the camera controller of the scene or the parent Block.



## Coordinate spaces

Currently `Makie` defines 4 coordinate spaces: :data, :clip, :relative and :pixel.
The example above shows te conversion pipeline for `space = :data`.

For `space = :clip` we consider `plot.converted` to be in clip space, meaning that `transform_func`, `model`, `view` and `projection` can be skipped, and `Float32Convert` only does a cast to Float32.
The x and y direction correspond to right and up, with z increasing towards the viewer.
All coordinates are limited to a -1 .. 1 range.

The other two spaces each include one matrix transformation to clip space.
For `space = :relative` this simply rescales the x and y dimension to a 0 .. 1 range.
And for `space = :pixel` the `camera.pixel_space` matrix is used to set the x and y range the size of the scene and the z range to -10_000 .. 10_000, with z facing away from the viewer.

0 comments on commit 67ebcf2

Please sign in to comment.