Add API support for adjusting camera roll angle, extend pitch angle range

[ssokol]

User Story

As a developer I can programmatically adjust the roll angle of the camera from 0 (level) to +/- 180° and I can programmatically adjust the pitch angle from 0° (looking down) to 180° (looking directly up) so that I can accurately represent the spatial orientation of the observer when displaying maps with 3D terrain.

Rationale

I am looking to use MapLibre in an aviation application to implement what is commonly know as "Synthetic Vision" - a feature which provides a virtual "view from the cockpit". To be able to accurately represent the this view, I need to be able to adjust the roll angle of the camera to match the roll angle of the airplane. You can see a similar system at a distinct roll angle here:

https://blog.foreflight.com/wp-content/uploads/2014/12/800x600-refined-terrain.png

From what I can find, the transformCameraUpdate callback currently allows programmatic control of bearing, center, elevation, pitch, and zoom (with pitch limited to 0° - 60°). Perhaps this could be extended to include the roll parameter and to accept pitch values from 0 (straight down) to 180 (straight up).

It might also be useful to allow for all of these parameter to be incorporated into the parameter objects for map.flyTo() and map.easeTo(). The way the Synthetic Vision app works, a server feeds situation data (location, altitude, track, speed, etc.) over a WebSocket at between 10-20 Hz. The handler for the WebSocket moves the camera using a string of very short duration flyTo or easeTo operations. Adding the ability to update situational parameters directly would simplify the process.

Impact

This won't have any real impact on users who only need a basic down-and-forward level view, but it will go a long way towards making MapLibre more useful for certain simulations, games, and situational awareness applications.

[HarelM]

Thanks for taking the time to open this issue.
In general, I think this can be a nice (although niche) addition to this lib.
The main caveat I can think of is which tile to fetch and show.
But feel free to push this forward.

[ssokol]

Thanks!

I'm slowly making my way through the library - long time Javascript user, first time Typescript user. I hope to be able to submit a PR, but it will probably need some help from those more fluent in the language. One question for you...

I haven't been able to find a way to manually set the camera altitude in feet or meters above median sea level. I kind of expected the easeTo and flyTo functions to have an altitude parameter, indicating the camera altitude at the end of the move.

Is there an API call to set the camera altitude independent of the zoom level? I see references to altitude in the camera.ts and map.ts files, but the altitude seems to be an output based on the current zoom level rather than something that can be preset programmatically.

[HarelM]

Zoom level is highly tide to altitude, it has deep historic roots and causes a lot of grief with terrain related stuff.
You can use cameraFromTo I believe to calculate some stuff, IIRC.

[ssokol]

Thanks. I'll look into that. I was just reading over the discussion of deterministic values on #4688 and it seems like the camera controls and associated motion methods may have come from the purely 2D world, where 'zoom' makes far more sense than 'altitude', and where pitch simply isn't a concept.

In testing some ideas I've discovered that the "center" parameter has very different meanings depending on if there is a pitch angle set. With a zero pitch (essentially a 2D map), center places the camera exactly where I expect it to. With a pitch value set, the lib appears to set the "gaze" or "focal point" of the camera at the "center" location, rather than the camera itself. Probably useful for some things, but challenging to use for a PoV application.

I wonder if 3D / terrain apps might be better off with a different camera control model. This would make center, altitude, bearing, pitch, and roll fixed and deterministic - these position the camera. The "gaze" or "focal point" is then determined by the zoom level (and optionally some additional parameters if needed). For my app, zoom will be constant throughout since it emulates what a human would see looking out the front window of an airplane.

Do you think there might be interest in a large bounty (perhaps funded by several participants) for a project that builds out a more 3D-centric "mode" or set of APIs?

[HarelM]

I don't have a good answer to the question about if there is an interest, and if someone would like to pay for it.
One thing to note is that one can switch from 2D to 3D and the expectation is to be at the same "position", so a concept where center means different things in respect to terrain may be problematic.
API-wise, I think Cesium has implemented something with unreal engine that allows looking from the ground up, might be interesting to look at what they did.
The concept of zoom is very much coupled to level of details (vector rendering) so that needs to be taken into account as well...

[ssokol]

My company would be willing to pay a bounty for a set of enhancements that make the 3D behavior meet my our case. I can see if anyone else in my industry is interested in participating.

On the center issue.... Is the current behavior (that center = focal point) intentional, inadvertent but accepted, or erroneous?

From what I can tell, giving the map a pitch angle and a center value positions the camera such that it is "looking at" the center rather than being positioned directly on the center. The actual camera position appears to be determined by some combination of the zoom value and the pitch angle.

If that's a behavior other users rely on, I wouldn't want to break it but would suggest creating a new cameraPoint parameter which is deterministic. If you set the cameraPoint, the lib calculates the focal point based on pitch and zoom. If you set the center, the lib uses the current behavior and determines the camera placement based on the center, pitch, and zoom.

Does that sound reasonable, or would it be better to change center control camera placement?

[HarelM]

Regarding the bounty, I would suggest to advertise it in our slack channel, as not many people follow the github issues.
In order to avoid breaking changes a parameter can be added to the map initialization to define a different camera behavior, something like immersive: true or something similar, once this is set thing will behave differently.
I think @cigone-openindoor had similar requirements.

• 360 immersive #1136

[ssokol]

Working on a proposal. Here is a link to the Google Doc for anyone who is interested:

https://docs.google.com/document/d/1ulpxmkuYbLlslq0mv1rUHCLCgN2ytKWWc-Iv5GPhFgA/edit?usp=sharing

[HarelM]

I've read the doc, thanks for sharing!
It would be great if someone could join the next monthly meeting to talk about it.
It's worth noting that the concept of zoom is integrated deeply with the style specification and styling in general when it comes to vector tiles, so I don't see a way to work around that. Having said that, I know it has its limitations when it comes to terrain and below sea level stuff, panning on heavy terrain scene etc...

Another approach would be to see which parts are align with current mapping needs and which parts may be exposed using some APIs and events to allow a plugin architecture for this specific case.
If the plugin is widely used and there is a demand to incorporate features from it into MapLibre we can always consider this in the future and allow faster implementation outside the main code base.
These are just initial thoughts.

[ssokol]

You're very welcome.

I would love to attend the next meeting but unfortunately I will be out on vacation on the second Wednesday of October. I can try to join, but chances are good that it won't work as I will be in the mountains of west Texas. I'm more than happy to provide any additional information that would help the discussion.

I suspected that zoom is very much a core concept since it's a core element of the tile path. In playing with the current library, I don't think that the "accuracy" or "verisimilitude" of the altitude is all that critical. Humans - even pilots - live in a 2D world: nobody knows exactly what things look like from a specific altitude. The important thing is relative altitude: the location of the camera relative to the surrounding terrain, ground placed objects (runways in my case), and sprites (other aircraft). The exact size of roadways and trees doesn't matter for my application.

I suspect someone more familiar with the library that I could readily come up with a heuristic that takes an MSL altitude (i.e. 392 meters above sea level) and translates that into a zoom level given the current field of view setting. I've tried to do something like that in my proof of concept and it almost works. If the camera placement issue was solved (i.e. the center and pitch angle determine the camera placement rather than the focal point) it probably would work.

In terms of the approach taken to building out the functionality - the plugin vs. core modifications - is above my pay grade. The plug-in route sounds quicker in the short term, but I don't want to end up with the plug-ins falling out of sync with the library over time.

Please let me know if you see anything in the document that seems unrealistic or too far out of scope for MapLibre. I don't want to do anything that would push the project off of its core mission.

[HarelM]

It's currently a very high-level definitions, I'll need to see a software design document to be able to say if something is relevant to be in the core code vs new API vs plugin.
In general, we try and make the public API stable so that people will know what breaks and when (major releases etc).
But it feels a bit early to talk about it. Let's see how this progresses first.

[NathanMOlson]

@ssokol @HarelM @Samarth1696

The camera rotation MapLibre is currently defined by 2 Euler angles, bearing and pitch. To define an arbitrary camera rotation, we need 3 Euler angles (or an alternate rotation representation such as quaternions). To implement this proposed feature, we need to make an early decision on how to represent the camera rotation .

Here is how the camera rotation is currently achieved:

1. Point the camera straight down, North up.
2. Rotate the camera about its Z axis by bearing.
3. Rotate the camera about its X axis by pitch.

Assuming we stick with Euler angles (which I think we definitely should for the public API, but maybe not for the internals), the two options we have to define the third Euler angle are

4a. Rotate the camera about its Z axis by roll (this puts the singularity at pitch == 0, camera pointed straight down).

or

4b. Rotate the camera about its Y axis by roll (this puts the singularity at pitch== 90, camera pointed at the horizon).

With either rotation representation, we can express any arbitrary camera rotation. Since the camera is virtual, the Euler angle singularity is just a mathematical issue, not a physical one. It will only cause problems when doing things like linear interpolation of Euler angles near the singularity (we currently do linear interpolation of Euler angles in easeTo() and flyTo()).

Option 4a is a better fit for @ssokol's use case. It is closer to the Euler angles traditionally used for aircraft, and puts the singularity at nadir, where @ssokol is uninterested. It also offers a more intuitive expression of the camera rotation, and would be a simpler extension to the existing code.

Option 4b would be a better fit for something like this: #1980. It would also keep the singularity away from nadir. Nadir feels like an strange choice for the Euler angle singularity in a mapping library that has traditionally been geared toward nadir maps.

Regardless of which choice we make, I think easeTo() and flyTo() will have to be re-written to avoid interpolation of Euler angles, in order to be completely generic.

IMO Option 4a is a better choice overall. I'd love to hear others' thoughts on this.

[ssokol]

As you predicted, 4a works better for me. In my use case, if the camera is pointed at nadir, the user has more important things to look at than the screen. (Though there are aerobatic maneuvers that can result in a straight down pitch attitude - briefly.)

Our current attitude and heading subsystem are geared for Euler angles, so that would be the easiest approach. Our AHRS internally uses quaternions but the output to display systems is just Eulers plus a few other values (rate of turn, slip/skid, g-load).

Do we take a chance on messing other users up by revising the functionality of easeTo() / flyTo()? Or would it just be the internals that change with the behavior remaining consistent?+

[HarelM]

It would be interesting to look at what choices they made in Cesium for this.
Creating a mathematical issue at nadir sounds like a footgun as this is basically the default behavior for a mapping app (the opposite of a flying simulator I guess), so I'm not sure what is the right course of action here.
It would be interesting to hear some more opinions...
Current implementation of methods like flyto and easeTo should remain the same assuming roll was not changed (i.e. default behavior), and I'm general, we need to look closely on behavior changes and decide case by case what to do and if we need to break it or not.

[ssokol]

Here's a link to a quick video walking through the 6 tasks associated with this enhancement, the bounty for each, and also some of the details of my use case:

[NathanMOlson]

Do we take a chance on messing other users up by revising the functionality of easeTo() / flyTo()? Or would it just be the internals that change with the behavior remaining consistent?+

My idea is to take the shortest path in rotation space from start to finish (instead of interpolating Euler angles). This would change the current behavior very slightly but in practice I expect it would not be noticeable, and in my opinion it would be "better". :)

It would be interesting to look at what choices they made in Cesium for this.

Cesium uses "aircraft Euler angles" (same as 4a but with pitch offset by 90 degrees so zero pitch means pointed at the horizon). The singularity is at nadir. https://github.com/CesiumGS/cesium/blob/6c2e520420b95bcb6c8eba0f02c76347cee1dd4b/packages/engine/Source/Core/Matrix3.js#L354

Cesium also does linear interpolation on Euler angles. https://github.com/CesiumGS/cesium/blob/6c2e520420b95bcb6c8eba0f02c76347cee1dd4b/packages/engine/Source/Scene/CameraFlightPath.js#L187 This means certain certain camera moves will be "ugly". But it's at least one data point that linear interpolation of Euler angles is "good enough". The types of moves that will be problematic ones where pitch crosses 0 (for example, flying from pitch = 1 to pitch = -1). MapLibre doesn't currently support those types of moves.

[NathanMOlson]

Here's a draft implementation of adding camera roll angle: #4771

You can play with it here, using the controls in the top right corner: https://nathanmolson.github.io/camera_roll

[wiesehahn]

the demo looks nice, I could imagine great interpolated flyTo animations.

However with this option its not possible to achieve similar behaviour as in #1980 (comment) (tilt around y-axis), or would this somehow be possible?

[HarelM]

I think it could be possible using the correct math, as this is basically the only rotation that is currently not supported.

[ssokol]

That looks fantastic, and is exactly what I had in mind. Well done!

[NathanMOlson]

However with this option its not possible to achieve similar behaviour as in #1980 (comment) (tilt around y-axis), or would this somehow be possible?

I've added 4 buttons to the demo to show the behavior near the singular point. Each button moves the camera to a specific rotation. Going from (-89.9 -89.9) to (89.9, 89.9) looks fine, but going from (-89.9 -90.1) to (89.9, 90.1) adds an extra spin (despite taking the shortest path in Euler angle space). This is caused by interpolation of Euler angles in easeTo().

This would be solved by changing easeTo() following the shortest path in rotation space, instead of the shortest path in Euler angle space.

https://nathanmolson.github.io/camera_roll

[Samarth1696]

I think using quaternions for internal rotation would be a better approach. We can provide the API such that it allows developers to choose the axis of rotation for the roll angle, either around the Z-axis (Option 4a) or the Y-axis (Option 4b). This would help solve issue #1980 and assist @ssokol with his project.

Do we take a chance on messing other users up by revising the functionality of easeTo() / flyTo()? Or would it just be the internals that change with the behavior remaining consistent?+

As per @HarelM ’s concern, we need to be careful not to disrupt the current implementation. We could convert Euler angles to quaternions (eulerToQuat) for internal rotation calculations whenever necessary and convert back to Euler angles (quatToEuler) for the output.
After exploring a bit, I think we could use the quat from gl-matrix?

Our current attitude and heading subsystem are geared for Euler angles, so that would be the easiest approach. Our AHRS internally uses quaternions, but the output to display systems is just Euler angles plus a few other values (rate of turn, slip/skid, g-load).

@ssokol Does it happen something similar on your side like I said above?

We can use some references from here:

• 3D Math Primer for Graphics and Game Development (This is a nice book; see Section 8.7.5)
• http://www.euclideanspace.com/maths/algebra/realNormedAlgebra/quaternions/slerp/

Feel free to correct me if I'm wrong, as I am quite new to 3D math.

[NathanMOlson]

I think using quaternions for internal rotation would be a better approach.

Agreed. Here's the change to easeTo() that uses quat. https://github.com/maplibre/maplibre-gl-js/pull/4771/files/ca8e5718bdbed4fdc99d1b497d06defe72d773b4..2e90e5e937141bcd41e745092ab811212be557cd#r1780266791

This change solves the issue demonstrated here: https://nathanmolson.github.io/camera_roll

You can see this change compared with the current behavior in this demo: https://nathanmolson.github.io/slerp/#12/47.27574/11.39085/0/1

[NathanMOlson]

We can provide the API such that it allows developers to choose the axis of rotation for the roll angle

I do not think we should do that. I think we should use quaternions where necessary internally, but expose a single Euler angle set in the public API (setBearing(), setPitch(), and newly added setRoll()). I think option 4a makes the most sense intuitively.

To make something like #1980 easier, we could additionally add setRotation() which accepts a quat or mat3, and let the user directly specify the rotation matrix without using Euler angles. This is something only advanced users would do.

[Samarth1696]

You can see this change compared with the current behavior in this demo: https://nathanmolson.github.io/slerp/#12/47.27574/11.39085/0/1

That looks nice as expected. I am not able to see any difference between those two. 🤔

To make something like #1980 easier, we could additionally add setRotation() which accepts a quat or mat3, and let the user directly specify the rotation matrix without using Euler angles. This is something only advanced users would do.

If you are able to do that then it's not an issue.

[NathanMOlson]

@ssokol Camera Roll is ready for review. I'm starting on decoupling camera location from centerPoint and extending the pitch angle range.

[NathanMOlson]

Initial ideas for decoupling location from center point are here: NathanMOlson#1

[NathanMOlson]

@ssokol @Samarth1696 Initial demo of camera-centric controls here: https://nathanmolson.github.io/camera-centric

Code here: NathanMOlson#1

This adds a new public function map.jumpToLLA() that allows you to set camera lat, lon, alt, and rotation directly. Internally, center point and zoom are still used to control the view. I explored using a different internal representation but decided against it.

In the demo, the buttons in the upper left call map.jumpToLLA(). This is a one-time repositioning. Using the mouse or other native controls changes the camera point as it previously did. When terrain is enabled, the map usually disappears at around 90 degrees pitch (sometimes a little more, sometimes a little less). I'm working to resolve that issue.

[NathanMOlson]

Camera-centric controls are close to finished, and there is now a draft PR with the proposed changes: #4851

[Samarth1696]

@ssokol @HarelM @NathanMOlson

Before proceeding with the implementation of motion control, I would like to gather your thoughts on the following proposal. Based on Steven's use case, I plan to create a method such as updateAircraftData, which allows developers to input various parameters to modify the camera's trajectory. This approach would benefit Steven by providing real-time camera motion. However, I feel that this method might not fully capture the immersive 3D experience we aim to deliver. We should think to offer users an good UX where they can easily control the aircraft by adjusting inputs like throttle, aileron (roll), rudder (yaw), and elevator (pitch). This would provide more control over the aircraft, accommodating multiple simultaneous inputs, as would be necessary for a game or 3D simulator.

I think updateAircraftData method could facilitate this functionality, but I am uncertain. I have been considering separating the 'motion_control' branch from 'simul-3D'. In the simul-3D branch, we would add additional physics such as drag force, g-force, etc.

I would like your feedbacks on this matter. We have two options to consider:

1. Implement Camera Centric as the initial plugin and allow users to choose between AircraftMode -> where they can use updateAircraftData to calculate motion -> and Simulator3DMode, where they can control the camera based on the simple inputs mentioned earlier.
2. Proceed exclusively with AircraftMode and not introduce Simulator3DMode.

This is a small demo of how simul-3D would look like:

simul3D-demo.mp4

It currently has some issues which I am working on, such as the pitch not exceeding 90 degrees, which would otherwise increase the elevation of the camera. Additionally, the camera undesirably enters the ground at the start of the motion(you can see in the demo), which should not occur.

[HarelM]

If the current support that was added (including pitch > 90 PR) is sufficient and you can create your code without using internal fields I think a plugin would be the right solution.
If there are still missing public APIs let's discuss those.

[NathanMOlson]

I think the aircraft simulation should be kept separate from the motion control outputs @ssokol has requested, and all of it should be outside MapLibre.

I think the outputs of your "Simulator3DMode" should be given as inputs to your "AircraftMode".

[Samarth1696]

If the current support that was added (including pitch > 90 PR) is sufficient and you can create your code without using internal fields I think a plugin would be the right solution.
If there are still missing public APIs let's discuss those.

One of the things I want to talk about is tile loading. You can see from the demo that many tiles were loading slowly or some were not loading(maybe due to max concurrent requests to the server) and I know the Motion was too fast but we can't expect what the users want. Can we do something to load future tiles? Like loading the tiles which are not in our viewport? This is mainly for Simulator3DMode.

I think the outputs of your "Simulator3DMode" should be given as inputs to your "AircraftMode".

Simulator3DMode functions differently from AircraftMode. The primary reason for separating these two modes is the calculation of delta time between camera updates. In Simulator3DMode, the delta time is taken into account between two frame requests, whereas in AircraftMode, it is taken between two input updates. Delta time is crucial, as it is a key factor in calculations for velocity, roll rates, lateral distance, and other parameters. Therefore, AircraftMode would provide the correct physics needed for @ssokol's use case or someone who wants to play with motion along the path.

I do not expect Simulator3DMode to achieve highly realistic physics compared to AircraftMode, but I will try to create more realistic physics as much as possible.

[Samarth1696]

@NathanMOlson @HarelM

Work on aircraft motion control:
https://samarth1696.github.io/motion_control/
The aircraft updates will stop after 9 seconds and you will see the camera is predicting trajectory on its own.

I have made some changes to the API with IControl as follows:

        // Initialize aircraft control
        const aircraftControl = new maplibregl.AircraftMotionControl({
            initialPosition: {
                lat: 47.27574,
                lng: 11.39085,
                altitude: 2000
            },
            cameraMode: {type: 'COCKPIT'}
        });

        map.addControl(aircraftControl);

Simulator 3D will go straight away as plugin but I'm uncertain about the best approach for the AircraftMode. Here's an example where the user can set an initial velocity and attitude to enable the camera to move forward:

        const aircraftControl = new maplibregl.AircraftMotionControl({
            initialPosition: {
                lat: 47.27574,
                lng: 11.39085,
                altitude: 2000
            },
            initialVelocity: {
                groundSpeed: 500,
                verticalSpeed: 100,
            },
            initialAttitude: {
                heading: 0,
                pitch: 85,
                roll: 0
            }
        });

You can test this example here:
https://samarth1696.github.io/testing/
We can change the naming conventions if we want this.

[HarelM]

@Samarth1696 can you clarify what the question is? I'm not sure I fully understand what input you require from my end...

[Samarth1696]

@HarelM I thought you said about setting AircraftMode as a plugin but I might have wrongly interpreted.. do you want this code to be a plugin or in the MapLibre?

[HarelM]

I think a plugin is the right place to put it. Does that answer your question?