Automation / Testing / Visual documentation framework

[ocornut]

(EDIT 2019: Even though this is an active topic (in 2019) note that this thread started a looong time ago and the early messages are not representive of where we are going now. When the new testing framework is a mature enough to be shared I'll close this topic and open a new one.)

(EDIT 2022: Released Dear ImGui Test Engine + Dear ImGui Test Suite
https://github.com/ocornut/imgui_test_engine)

Looking for someone who would be interested in working on that, probably under my guidance.

Mentioned it in #259

Add a framework for generating screenshots/animations for given pieces of code. This is probably going to be done along with the "testing" framework. I would like the ImGui documentation to be a very visual, well organized listing with C++ code on one side, screenshot (or animated GIF) on the other side and tags + commentary. I think this will easily convey a lot of ideas and tips about ways to use ImGui.
Update: I've created some helper to help me take manual screenshots and its been very helpful in the past few days. This will probably evolve into such framework.

The idea would be have a separate test application, could be or not on the same repo, that would A/ generate screenshots/gif + code markup output for the wiki and B/ defacto run tests.

So we could describe those documentation elements in the code, the framework would execute the test, simulate inputs to interact, take screenshots or gif, crop them as desired, parse its own .cpp file to retrieve the code (which itself could have marking in comments to hide some lines) and generate the help section that would demonstrate ways to use ImGui by showing the code and the screenshot. We would have hundreds of short examples, grouped, chaptered, tagged, indexed.

I was originally primarily interested in the value of documentation but I think the overlap with testing will be enormous and natural. Just the act of generating the documentation from runtime will be an act of testing. The testing wouldn't be unit tests at ultra fine granularity but rather functional tests (at least this is what I think they are called in the real world?): say, move mouse to corner, click and drag. We can compare screenshot, regression easily caught. Tests would be merely the unpublished part of the documentation system. Any tricky area of the code or fixed bug can be subject to a test, etc.

It's not really a trivial task to set all that up and I think it'd be highly valuable to the project. Realistically I don't think I'd be able to tackle by myself this year, so looking around if someone is interested to contribute :)

The person would be expected to be proficient in writing terse, optimal C++ and willing to adopt the general programming style of ImGui.

Thanks!

[ocornut]

PS: to clarify even if we could aim to have lots of it in a separate file, that would still involve touching a lot of the core library, and some of the changes will probably lead to imgui's own public api development.

[emoon]

This sound like an awesome idea. I have thought about this before as I want to document the UI API in my debugger. The API is close the exactly the same but with a different naming scheme but that is very minor. I would likely try to use this as well (given permission of course) in a slightly altered form.

[emoon]

I'm bit interested in helping out with this.

I wonder if you have some ideas on how to do it? That being said I don't have lots of time doing this but I want to help out at least.

[ocornut]

Thanks Daniel and sorry for not answering to this earlier. :)
Some out-of-order notes on top of my head.

That picture is essentially showing the intent:

Except it's all wrong because:

• the code is embedded in a picture
• it's just showing random unrelated stuff
• the example isn't complete and lack context
• and mostly, it was painfully and manually authored

I think a first step would be to manually draft more example documentation. Which would naturally lead to creating the framework for this. It would probably embed something like the screenshot helper here https://github.com/ocornut/imgui/wiki/screenshot_tool

How should the documentation be presented? When and how can we be skip context? Then we figure step by step out how we can make this work easier and more automated. Believe it or not I feel pretty bad with my English when it comes to presenting things in a consistent manner so I could do with some help :)

I am not too concerned about tackling the testing part too explicitly at first. ImGui doesn't really crash so tackling this possibility in an automated manner isn't an important priority ihmo. I think for a start the documentation itself can be defacto testing.

Overview style vs Single-function/symbol style

I think there would be more value in providing chapters/article about series of functions rather than individual functions. There's nothing more frustrating that documentation which have a line about each functions but you never get to understand the whole picture. So I would rather have a chapters called "Menus & Popups" and say, BeginMenu() EndMenu() are documented together, than separate entries. It's still possibly to have separate entries for every functions as the second half of those chapters if we feel it makes sense. As the output will be automated we will be able to build a full index anyway.

Also, the documentation shall be seen as a general "usage guide" which include suggestions, tips, making uses of several functions to benefit from a certain pattern, etc. so I feel it is nice to not overly focus on single function/symbols.

Note about repo size: I would prefer to keep the main repo small because any commit ends up in user clone. So if we end up uploading pictures they could be on the Wiki repo (which would make sense for the documentation anyway) or we create a imgui-docs repo just to host the data

[ocornut]

Here's a quick example I just hacked
EDIT and here's the code: https://gist.github.com/ocornut/4d9001336d07a6818f96
Code is pretty bad/hacky/shortsighted but just giving the general idea of where things might go. It only generate the pictures for now. It should eventually generate a full .md file with text and links to screenshots?

---

Buttons

bool Button(const char* label, const ImVec2& size = ImVec2(0,0))

Button() returns true when it has been pressed

    if (ImGui::Button("Save"))
        MySaveCode();

Button() can takes an optional size, set individually for each axis.
A value of zero sets the size automatically based on the label.

    ImGui::Button("Click");
    ImGui::Button("Click", ImVec2(0, 0));

A positive value sets the size

    ImGui::Button("Click", ImVec2(100, 0));   // Specify width
    ImGui::Button("Click", ImVec2(0, 100));   // Specify height
    ImGui::Button("Click", ImVec2(100, 100)); // Specify width and height

A negative value align to the edge of the window or current contents region

    ImGui::Button("Click", ImVec2(-1, 0));    // Align to right edge
    ImGui::Button("Click", ImVec2(-20, 0));   // Align to right edge 
    ImGui::Button("Click", ImVec2(0, -1));    // Align to bottom edge 

---

[emoon]

This sounds good.

I also would like to separate the code examples into separate files that may or may not be included when generating the doc. That way it's possible to have allow others to use the documentation but having slightly different syntax (for example in Rust or C wrapper case) but still use the guts of the documentation.

Still the generated doc can include samples with the code in question but selectable in some way.

Also using markdown for the documentation would be great.

[ocornut]

Yes ideally it would become a set of helpers that can be used in different contexts, locally or for languages ports. It could potentially even be useful for people to document their own imgui-based helpers or even post repros here :) but we'd have to see how it evolve. Shorter term I would just suggest tackling the basic problems: parsing source code, outputting to markdown, adapting the system so that it works in more contexts. Some context may include or exclude the creation of a window, some context may need to hide some code, etc. I could create a imgui-doc repo right now and grant you access if you want to fiddle with it together and we can always scrap the repo if it ends up back inside the main repo later

[emoon]

I was thinking of doing it the other way around.

Parsing markdown (with inline or ref code) and generating code to be executed and then screenshoted. That way one doesn't need to parse source code as markdown is likely way easier to parse.

Thoughts?

[ocornut]

I would guess it is easy to parse source code as long as we have known markers and don't try to support edge cases, and it's nice to write and iterate/run actual code directly in place, rather than indirectly via a MD layer. But either way would work.

I semi expect the code to include various markers/oddities. (e.g. a marker to request "hiding a line" or hiding unnecessary scaffolding from from the user-visible output) so even if MD is used as source, we would have to regenerate a different MD either way. For example for a particular series of example demo we may need to setup certain conditions but maybe the setup is only visible once (or never if it's not really important in context).

[emoon]

Yeah if we use markers then it should be fine.

[ocornut]

Created an empty repo for code https://github.com/ocornut/imgui-doc
I don't mind keeping the discussions here as the other repo may or not last

[emoon]

Thanks! I will give this some more though and come up with a suggestion.

[emoon]

Sorry for the lack of updates on this one. I will try to get it done this week.

[ocornut]

No worry! Been myself going to a crazy patch of work from which I hope to escape soon :) Thanks for looking into that.

[emoon]

Still haven't got around to do anything more about this :( If anyone else wants to help out with that would be great but my plan is still to look into it even if it will likely ta a while longer than I hope.

[RobertoMalatesta]

@emoon see my comments on #693
I'd appreciate your opinion.

--R

[RobertoMalatesta]

Hi @ocornut , @emoon,
it's not clear to me (and maybe only me) what the depicted system will try to achieve.
It seems great, fantastic. On paper.

I semi expect the code to include various markers/oddities. (e.g. a marker to request "hiding a line" or hiding

How badly having new macros or directives will impact the terseness & understandability of code?

And, first and foremost, which ETA?
(you know, I'm as curios as a cat, and I'd like to be still alive, the day this systems comes out)

Set aside the automatic testing issue (up until someone eventually solves the Halting Problem),
I wonder when this system will be available.

Wouldn't be more realistic --that's a nice name alternative for IMGUI: RealisticUI btw!-- to use already existing and not-too-complex-to-learn tools ?

I will state my proposal in a concrete way:

1. adopt DoxyGen as a simple, unobtrusive way to document and map code (up to graphs and UML charts)
2. implement an official emscripten port of imgui (I would offer mine, but it's two years old and @floooh one is definitely the best.
3. create an official way and directory to create extensions and plugins for IMGUI and document it as a priority issue.
4. drop some basic examples there along with some templates to creating new ones
5. create a script to build and generate a page containing source code on the left and running, live example on the right using 2.
6. link doxygen autogenerated documentations to examples at point 4 (automatically, with tags inserted in normal source comments)

That's basically all.
No new macros that will turn IMGUI into BOOST, no new system to create falling victim of NIH Syndrome.

--R

[ocornut]

Doxygen is just not good enough and won't provide a testing infrastructure. Testing is the first need. There's no ETA, imgui is a hobbyist project. The aim is to make something great nothing less. If we were low-aim realists we'd be stuck with QT forever. I may have more time for ImGui from next January. It will also be an ever increasing importance to many big ImGui users (game studios) that those testing systems get implemented so perhaps they will have more incentive to help.

How badly having new macros or directives will impact the terseness & understandability of code?

Those would be in the testing code, not in core imgui.

Your proposal doesn't solve the testing requirements, and pardon me but doxygen is quite obtrusive. Many imgui users would be freaked out if we doxygened the code. This library is built in such a weird/specific unorthodox way that classic doxygen just isn't so appropriate.

Right now I think the documentation isn't that bad for experienced programmers. As strange as it seems, it is not the current priority to make imgui accessible to less experienced programmers. Just because we don't have the bandwidth/resources to handle influx of users and solve every problems. It needs to mature slowly. And I think the library isn't finished. It may take one year, two years until it feels finished. You are right on many points, we should improve extensibility, improve examples, but that's all lots of work it doesn't happen magically. Right now I think the focus should be testing, adding useful features, improve code sharing, attract experienced programmers who can help. Accessibility will slowly improve :) I just only have so many free hours, need to make choices. Sensible contributions from imgui users are most often welcome.

[RobertoMalatesta]

HI Omar, and thanks for the time taken for giving me a detailed answer.

doxygen is quite obtrusive. Many imgui users would be freaked out if we doxygened the code.

I'm not trying to sell you something, so it's OK for me.

Testing is the first need.

one thing I left out was:

7 . implementing a web-based playground where any developer could fiddle in code and experiments:
a small app template inside a page, with a compile and run option. That way proposing examples and components or signaling bugs would be easier and done with a standard approach.

Right now I think the documentation isn't that bad for experienced programmers.

Agreed, but if something takes me T instead 10*T I prefer the first route nevertheless.
And documentation is more than a quick route for n00bs, it helps define and formalize the API pact between who's using it and the developer of the component.

we should improve extensibility,

Really. More than one year ago I implemented a basic chart wiget to do pies and histograms, along with a calendar control. It was a day's hobbyist work as a test of what IMGUI could give me.
The code was cutty-pastish, but it worked.
Then I thought for a while how to clean them and how to do a push on github.
But everyday's work took over and those components went forgotten onto one disk somewhere.
Pity since those snippets could have been useful someone else to start over with.

In different github repos I see interesting components developed on modified forks. A pity that the lack of a common extension system does not allow us to summon them all.
This should be priority one,

--R

[dougbinks]

I'm putting some thoughts on this issue here after a brief discussion on twitter:

So there are two major parts to this, an example code document generation framework and a regression testing framework. The idea is to make these work together so the examples are also tests, making it easier to maintain.

Documentation Framework

The documentation system seems doable in a fairly straight forward way using standard C++ and macros. I think the requirements would be something like:

1. Minimally intrusive to example code.
2. Not required to be present in a normal build of the examples.
3. Easy to use.
4. Lightweight, pref single .h/.cpp pair.
5. Automated.

For 1 I would use something like this example but with IMDOC_BEGIN not creating a window as I think this obfuscates the original source code. So I would have something like:

IMDOC_BEGIN("Button: Default Size");
IMDOC("Button() can takes an optional size, set individually for each axis.");
IMDOC("A value of zero sets the size automatically based on the label.");

    ImGui::Begin("Button: Default Size");
    ImGui::Button("Click");
    ImGui::Button("Click", ImVec2(0, 0));
    ImGui::End();

IMDOC_END();

Note that this approach makes the ImGui::* code able to be pasted into code not using the doc system, and more easily searchable.

Requirement 2 is simple enough using macros, as the regular imgui include can define the macros as producing nothing.

To be easy to use means that the output should be a .md file and a directory with the required images in it. Preferably that file would contain all the documentation, so we may need macros to be able to include source .mds for long preamble you don't want in the .cpp file.

The lightweight part is debatable, as leveraging scripting tools or lang tooling might be more powerful, however it's doable and probably faster to get the basics up and working.

To be automated should mean that simply running the code gives the output with no need for user input. This would mean no default closed windows, or that the doc system can control whether a window is open or not.

Regression testing Framework

This is where things get tricky.

I think I understand the objective - inject input and capture screenshots (we may want to capture the entire back buffer to ensure we capture changes in imgui window position) then compare against prior data and flag changes to some easily viewable output.

The following code is for a demo which doesn't open new windows and where we want to test widgets in turn (range based, euto etc for brevity):

test_setdrawfunc( demo );
auto windows = test_get_windows(); // runs demo function, gets list of windows

for( auto window : windows )
{    
    auto widgets = test_get_widgets( window ); // see below
    for( auto widget : widgets )
    {
        test_hover( widget );
        test_click( widget );
        test_doubleclick( widget );
        test_entertext( widget, "foobar 99.0 []&$" );
        test_clickdrag( widget, 0.1f, 0.0f ); // units in size of widget
        test_clickdrag( widget, 1.5f, 0.0f );
    }

    test_drag( window );
    test_move( window );
    test_size( window );
    test_close( window );
}

test_get_widgets( window ) would work by iterating the mouse coords over a window and getting all different HoveredId's from the ImGuiContext.

Each test function would likely need to reset state, run the demo function for a few frames, add the relevant input whilst running the demo and capture & output pngs and any required data.

Code would then need to be written to compare pngs and generate a table of results.

[RobertoMalatesta]

I'm still skeptical on this but I'm willing to give full support to your effort.
@ocornut is our Obergruppenführer since it worked from day one to now-
--R

[ocornut]

Still working on this in the background.

It is surprisingly difficult to get robust results given varying starting state. Normally most tests should run on a predictable starting state so this shouldn't be an issue, but making the high-level functions extra robust will help the system to be of use for user application, maybe even end-user exposed macros. (Still injecting mostly actual mouse/keyboard inputs.)

[elect86]

Hi Omar,

is this code already available somewhere?

[ocornut]

is this code already available somewhere?

It isn't. I can give you access to a repo but for now it is experimental and the API will keep changing for several months.

[elect86]

Yes, but only if I may start porting it on the jvm, otherwise I'd just wait

[ocornut]

It's way too experimental/early to be ported.

[ocornut]

Still working on this in the background (still very useful during development, stil too early to publish).
Thanks to @rokups everything is now nicely wired to continuous integration systems via GitHub Actions (beta feature open to everyone), so we now build several examples on many platforms + the testing suite also runs on its own repository.

The core imgui side of things should be visible here: https://github.com/ocornut/imgui/actions

[gavlig]

Hello, may I request access to the said repo with testing framework as well? I understand that its api is unstable but it would still be better than hacking internal application state for tests.

[ocornut]

@gavlig Please e-mail me about this.

[adamierymenko]

I found this thread trying to figure out of this project supports screen readers for the blind, which requires some kind of accessibility API. I can't figure out the answer to this but I assume this is related. Anyone know? Should go in the FAQ. If not is this planned?

[ocornut]

@adamierymenko On my side this is not planned at the moment. There's a little bit of overlap with the work done for this issue, but overally the bulk of the work would be to figure out and interact with the dedicated OS accessibility api which are particularly complex to deal with. Someone more experienced with accessibility need would be more a adequate fit to look into this.

[elect86]

The CI integration is the cherry on top of the cake..

May I have access, Omar? I'd like to start playing around and see where it goes..

If this goes well, it will greatly help me to sync with your changes with much less pressure on my shoulders about introducing bugs/regressions, and also to find out which is the commit to blame when a test is failing (without me checking out again and again randomly commits to find it)

I read here it can even run headless..! This is really nice..

[ocornut]

A GIF of automation doing regression testings on some subtle tree node behaviors
(testing combinations of: open on arrow flags, double click flags, with/without multi-select api, treenode as drag source, holding drag payload to open node, exacting reaction frame (some need to react on MouseDown other on MouseUp etc.):

[MouriNaruto]

I wonder to suggest imgui that add the support of MSAA or UIA for Windows platform because it can help lots of people.

[elect86]

Hey @ocornut,

may I ask you also access for one collaborators of ours (@Sylvyrfysh)?

[pthom]

Just a quick suggestion, which I hope is not too strange. imgui.h could be used as a source in order to generate external doc.

My idea stems from markdown preview enhanced, which enables to add diagrams, equations, table of content, etc to markdown.

Anyhow, I have made some test in another project, and I can obtain doc that looks like this: api.md

With a doc "source" which looks like this: api.src.md

A where a documented .cpp or .h file might look like this: source file with embedded markdown

And the doc generation script could be inspired from this: script

[ChiefyChief23]

@ocornut Hey, I was wondering if I could get access the repo with the testing framework? I've recently got into ImGui over the last few months and want to try and contribute to it too

[canatella]

Hello, would also be interested in testing the testing framework.

[dashandslash]

@ocornut I would love to try out the ImGui automation testing framework, could you please add me to that repo?

[ocornut]

If interested please e-mail me and specify: if it is for a personal or work project, share a bit about nature of the project and how you envision automation could be useful for you. We can't let everyone in just yet but working on it.

[dashandslash]

If interested please e-mail me and specify: if it is for a personal or work project, share a bit about nature of the project and how you envision automation could be useful for you. We can't let everyone in just yet but working on it.

Hi Omar, I sent you an email few weeks ago on this matter, let me know if it's still possible to join the private repository for UI Automation Testing.

Cheers
Luca

[jacobfriedman]

Bump- is this still active @ocornut / @dashandslash ?

[ocornut]

Released Dear ImGui Test Engine + Dear ImGui Test Suite
https://github.com/ocornut/imgui_test_engine
Closing this old very-general issue., though I have no doubt we also have things to do core Dear ImGui side (e.g. #3949).