New IO event API: io.AddKeyEvent(), io.AddMousePosEvent() etc + full range of ImGuiKey values w/ backward compatibility

[ocornut]

Before we merge into master, this thread is asking for curious testers and feedback.

Temporary test branches:
https://github.com/ocornut/imgui/tree/features/extra-keys-and-key-event
https://github.com/ocornut/imgui/tree/features/extra-keys-and-key-event-docking

Moved to #4921 now that things have settled.

TL;DR;

The TL;DR; is we are switching from referring to keys using native/backend-dependant indices ImGui::IsKeyPressed(MY_ENGINE_KEY_F1) / ImGui::IsKeyPressed(GLFW_KEY_F1) to referring to keys using a full-range standardized ImGuiKey enums ImGui::IsKeyPressed(ImGuiKey_F1).

This will allow writing various features that we couldn't before, and sharing code that's using keyboard while staying backend-agnostic (this was problematic before and a hurdle for people writing code to share).

The challenge was getting that transition done elegantly without causing backward compatibility issues, nor introducing complications or e.g. warnings.

Details

Initially the ##2625 PR by @thedmd was about adding ImGuiKey enum values, trivial at first glance. We discussed it privately for a long time. I had two concerns: existing API made using those keys actually tedious, and we didn't have a way to distinguish translated and untranslated keys (the earlier would be desirable for shortcut), if we start exposing new API (e.g. for shortcuts) without allowing that distinction people would write code that wouldn't localize correctly to varying keyboard layout.

Separately we worked on designing a new IO API that would allow handling of trickling events (for e.g. very low framerate), based on the io_input_queue experiment. Good progress has been made on that (but it got a little more involved than expected).

What we've done for now was to combine the extra-keys + reworking of IO API solely for keyboard keys at first to stay focused.
This was generally the work of @thedmd which I followed/feedbacked and help finish.

Among other things IHMO one of the nice breakthrough was coming with a scheme which is support transparent backward compatibility without adding new API entry points. We also solved fairly elegantly the little puzzle of allowing old backends to work as well as new ones. Technically this is mostly a base for us to build new features and its not taken advantage of yet (but user can use it). I think we will try to make some more use of it before declaring the change valid.

Changelog: Reworked IO keyboard input system. (#2625, #3724) [@thedmd]

• Added io.AddKeyEvent() function. Obsoleted writing directly to io.KeyMap[], io.KeysDown[] arrays.
• Amend: Added io.AddKeyModEvent() function, obsoleting writing directly to io.KeyCtrl, io.KeyShift etc.
• Amend: Added io.SetKeyEventNativeData() function (optional) to pass native and old legacy indices.
• Added full range of key enums in ImGuiKey (e.g. ImGuiKey_F1).
• Added GetKeyName() helper function.
• Obsoleted GetKeyIndex(): it is now unnecessary and will now return the same value.
• All keyboard related functions taking 'int user_key_index' now take 'ImGuiKey key':
  • IsKeyDown(), IsKeyPressed(), IsKeyReleased(), GetKeyPressedAmount().
• All backends were updated to use io.AddKeyEvent().
• Backward compatibility:
  • Old backends populating those arrays will still work for a while.
  • Calling e.g. IsKeyPressed(MY_NATIVE_KEY_XXX) will still work for a while.
  • Those legacy arrays will only be disabled if '#define IMGUI_DISABLE_OBSOLETE_KEYIO' is set in your imconfig.
    In a few versions, IMGUI_DISABLE_OBSOLETE_FUNCTIONS will automatically enable IMGUI_DISABLE_OBSOLETE_KEYIO,
    so this will be moved into the regular obsolescence path.
• Transition guide:
  • IsKeyPressed(MY_NATIVE_KEY_XXX) -> use IsKeyPressed(ImGuiKey_XXX)
  • IsKeyPressed(GetKeyIndex(ImGuiKey_XXX)) -> use IsKeyPressed(ImGuiKey_XXX)
  • Backend writing to io.KeyMap[],KeysDown[] -> backend should call io.AddKeyEvent(), if legacy indexing is desired, call io.SetKeyEventNativeData()
  • Basically the trick we took advantage of is that we previously only supported native keycode from 0 to 511,
    so ImGuiKey values can still express a legacy native keycode, and new named keys are all >= 512.
• This will enable a few things in the future:
  • Access to portable keys allows for backed-agnostic keyboard input code. Until now it was difficult to
    share code using keyboard accross project because of this gap. (Add extra keys #2625, Help with keyboard handling and catching some keys #3724)
  • Access to full key ranges will allow us to develop a proper keyboard shortcut system. (Shortcuts API #456)
  • io.AddKeyEvent() include native keycode/scancode which will later be exposed to access translated/untranslated keys. (Support for untranslated key codes #3141, Keyboard layout ignored for keybinds #2959)
  • io.AddKeyEvent() will later be turned into a trickling IO queue (for all inputs) to handle very low framerate better. (Fixed same-frame click/release bug #2525, Inputs are missed when FPS is low #2787, Collect input counts over one frame to avoid missing inputs within the same frame #3383)

Desired feedback

We'd like people to try this.

• Does this makes sense?
• Try to update.
• Does it work well with your OLD standard backend?
• Does it work well with UPDATED standard backend?
• Does it work well with your unmodified custom backend? Is updating your backend to the new io.AddKeyEvent() obvious? If you got issue can you shared them?
• Do you get compilation errors or warnings?
• Do you have lots of code referring to native keycodes? Is it easy to transition it? Try enabling IMGUI_DISABLE_OBSOLETE_KEYIO
• Are all the keys you need available in ImGuiKey, and correctly mapped in your backend?

This is how backends were updated:

• GLFW: 09b03a6
• SDL2: 8224095
• Win32: f58aa22 (extra complications because raw Win32 has oddities with some keys and modifiers)

[jokteur]

So I have tested this new event feature on my old project (https://github.com/jokteur/BM-Segmenter). I haven't touched the code in months, so first I had to make sure that it worked with last version of ImGui (1.86) with the docking branch enabled.

Configuration of project

I am using the glfw backend with OpenGl3. I am working with docking and multi-viewports enabled. Last working working version: 1.86. I am on windows 10, working with Visual Studio.

Testing of the branch

I checkout imgui to the branch https://github.com/ocornut/imgui/tree/features/extra-keys-and-key-event-docking. It compiled without problems, but the first problem I encountered when opening the program was:
Assertion failed: g.IO.KeyMap[ImGuiKey_Space] != -1 && "ImGuiKey_Space is not mapped, required for keyboard navigation.", file ..\..\..\thirdparty\imgui\imgui.cpp, line 8236
This is because I had ImGuiConfigFlags_NavEnableKeyboard enabled. I suppose that the spacebar will be correctly mapped in the future. So I deactivated this option for now.

Now, the program works, and I don't have any event related problems to report. I can type any characters in text inputs no problems so far.

Testing of GLFW backend in 09b03a6

I tried using the latest update in 09b03a6 (still with ImGuiConfigFlags_NavEnableKeyboard disabled). I had this compilation error:
Line 326 in imgui_impl_glfw.cpp: 'ImeWindowHandle' is not a member of 'ImGuiIO'.
Commenting this line (predictably) renders the multi-viewport useless.

Things to test

I am using a custom module for handling shortcuts (see here and here ; the code is a mess, sorry). This module allows me to do global shortcuts, local shortcuts (which override global shortcuts), timed shortcuts, ... However, I am using GLFW keycodes to handle the shortcuts. I would still need to rewrite this module using the new ImGui key events.

[ocornut]

Thanks for testing! The first issue was fixed this morning in the main branch (but didn’t backport the fix in the -docking one, my bad). The second I believe relate to code that is different between master and docking branch (the IME stuff) which I forgot to finish reworking. Changes were needed to make IME work on multiple viewports.

[ocornut]

I tried using the latest update in 09b03a6 (still with ImGuiConfigFlags_NavEnableKeyboard disabled). I had this compilation error: Line 326 in imgui_impl_glfw.cpp: 'ImeWindowHandle' is not a member of 'ImGuiIO'.

Off-topic: your message prompted me to do the small Platform IME refactor that caused some backends to be incompatible between master and docking. This is now all done. (starting from 3a90dc3 on master + small other features, and the docking 704ab11), so this is also fixed.

[ocornut]

Added a few back and forth this has been tweaked, improved, merged and applied to all backends!
Thanks @thedmd !

[caxapexac]

Thats very cool update but:

What about mouse keys to keyboard and vice versa (I mean to enable you writing shortcuts (or something based on input) device-independent). AFAIK unity engine includes mouse button into keymap (https://docs.unity3d.com/ScriptReference/KeyCode.Mouse0.html) so can it be the case?
Another thing is where you need to emulate mouse using keyboard so it could be useful here too

[ocornut]

Do you use and need those features ?

[caxapexac]

Do you use and need those features ?

@ocornut

If you are adding custom keymap ability to your app then you are assuming user can bind command either to ctrl+v (other world) or rmb-click (terminal specific) or mmb-click (realistic scenario) and you dont wanna check every case or storing keybind as sick union of key+mousekey. Honestly I think that input device unification can make archtecture much simpler and not to break backwards compatibility (old mouse checks just would be redirected to new enum inside keyboard space

It is sick to see how people are installing third party apps (ref: xmouse button control) on top of everything to get such simple stuff like binding something to mouse

One more case I've just remembered is mouse buttons 4+ (which are really used for shortcuts most of times. And again people are having to use external solution inside most of the apps)

As for me, I've used KeyCode.MouseX in unity3d many times, especially for key binding and it I've found it pretty comfy. Unity has 4 ways of handling mouse input and KeyCode way was the most usable for me

[Jiboo]

956e030 looked like it worked nicely without any code modifications on my custom backend.
Adding IMGUI_DISABLE_OBSOLETE_FUNCTIONS introduced a compile error regarding KeyPadEnter renaming.
Adding IMGUI_DISABLE_OBSOLETE_KEYIO obviously introduced way more problems, but the move to the new AddKeyEvent API was straightforward.

This was probably already discussed, but couldn't find it sorry, the keysym names are custom or based on some convention?
"Meta" may be preferred over "Super" (see https://www.w3.org/TR/uievents-key/#key-Meta), but wikipedia and other sources contradicts this, so dunno.
Here's my mapping of symbol names for linux's input.h vs x11 vs imgui.

[thedmd]

"Meta" may be preferred over "Super"

ImGui already used io.KeySuper and ImGuiKeyModFlags_Super. It was natural to name key after that for consistency. In practice it map to Cmd/Windows/Super keys depending on the platform.

[ocornut]

Moved to #4921 now that things have settled.

[ocornut]

IMPORTANT: If you have updated backends between January 10 and January 27
We have made a breaking change to the new IO api c906c65
io.AddKeyModsEvent() was replaced with normal calls to io.AddKeyEvent() with ImGuiKey_ModXXX values.

All backends have been updated. This only applies to you if you updated standard or custom backend between January 10 and January 27. If you use standard backends, simply update. If you use a custom backend change code above.