All-up Testing Goals/Priorities

[michael-hawker]

Related to #3

Testing Goals

We want to make testing approachable and easy for our developers. We've learned from the main Windows Community Toolkit repo that we can get a lot of flexibility in testing our UI features with the Unit Test infrastructure and the VisualUITestBase helper. This has allowed us a lot of depth in exercising UI without much added complexity to a standard Unit Test.

Ideally, we can have a single test (as written) run across one or more target frameworks to help us validate our extra platform targets. UWP and the WinAppSDK are our main priorities, next coming WASM, and then another like Android would probably be the easiest.

UI Testing has been trickier. Our current setup is pretty complex and a bit fragile. It'd be interesting to learn if there are other options out there that may simplify this process.

Unit Tests (Majority of our tests)

• P0: Write a Single Test
• P0: Run with both UWP and WinAppSDK? without modification (can use a separate head if needed)
• P1: To Investigate: Can we get 1+ Other Platforms (WASM, Android...) to work?

UI Tests (~5-15% expected of needs I believe)

• P0: Write a Single Test
• P0: Run with both UWP and WinAppSDK?
• P3: Run against another platform...

Also

• Integrate tests into the CI process
• Have easily readable test results (& badges?)
• Update test projects to use shared props

Summary

I think if we can at least get to a point where a single unit test, as written, runs across UWP, WinAppSDK, and at least one other platform, and a UI Test that runs on UWP and WinAppSDK. That should cover 95+% of our testing needs/requirements.

As long as adding tests is simple and easy to test locally (and in the CI for validation), then that should be the main thing. We want to lower the barrier to writing tests so we can write more tests to help us maintain code. This enables us or others to better manipulate existing code to fix issues without worry of breaking core functionality when the original code may not be theirs.

[mrlacey]

Research progress to date:

For unit tests

• Having a single test run within the context of a UWP & a WinAppSDK app (via separate heads) is easy.
• Doing the same in another project type (I've only looked in detail at Android & WASM) isn't possible without significant investment in custom text execution code.
  • The only ways that unit tests are run for Xamarin or native Android apps is on a local dev machine (e.g., the PC) against mocked-out platform abstractions. I don't think doing this would give us anything beyond having and running the Windows-specific tests.
  • Running unit tests in the context of an executing WASM app would mean creating something new to ensure the tests were compiled and executed correctly. I'm not sure that creating what would amount to a new, custom testing framework is appropriate. A better approach to test on non-Windows platforms (including, but not limited to WASM) will be to use UI tests for this.

UI Tests

• Running against UWP with WinAppDriver - easy
• Running against WinAppSDK with WinAppDriver - still investigating but should be straight forward
• Running against another platform - still investigating but have encountered several gotchas:
  • Uno.UITest (Helpers -- in all samples and templates) uses NUnit rather than MSTest (Presumably so it's not just limited to running on Windows)
  • Uno.UITest doesn't (yet) support testing on Windows.
  • Uno.UITests currently feels like the best solution to base this on.

Points for further discussion/clarification:

• How much time is it worth spending investigating running tests beyond Windows?
• Do we need to stick with MSTest for all tests? (all toolkit tests currently use this.)
• Could we consider using MSTest for unit tests and NUnit for UI tests? (so we could take advantage of Uno.UITest)
• Is it appropriate to invest in adding Windows support to Uno.UITest? with or without using MSTest? (I have an idea of what's involved but not yet dug deep enough to give an estimate of the work involved.)

[michael-hawker]

Thanks for the update @mrlacey! Lots of good info. @Arlodotexe saw that James posted about their new Uno testing solution here: https://twitter.com/jamesmcroft/status/1508042090013462529?t=_c6BR9vH20ag12T5Ee7mRw&s=19

Do you think it'd be worth taking a look at that to compare to the Uno.UITest solution? And then we can discuss which approaches make the most sense based on our original requirements?

[mrlacey]

Here's what my investigations regarding UI testing have identified.

There are two options for free cross-platform testing tools that support Windows, Web, & Android.

• Uno.UITest
• Legerity.Uno

Both options offer serious challenges to consideration and use.

Challenges to using Uno.UITest

• Currently has no Windows support.
• Tool maturity and stability are low.
• Recent development investment is low.

Challenges to using Legerity.Uno

• Requires more configuration and setup.
• Still a very new solution in the preview stage.
• I have been unable to get Android-based testing to work,
• Windows-based testing relies on WinAppDriver, which is unsupported, has an uncertain future, and has many known issues.

Recommendations:

• I do not think either of the above options is currently appropriate for use for our purposes.
• Based on the small number of UI tests we expect to write, I think that attempting to use any cross-platform testing solution will be a poor use of time as there will be a lot of setup, configuration, and support for the associated infrastructure. This, combined with the above's reliability issues, is a good argument for doing something else.
• With the future of WinAppDriver so uncertain, I think we should continue using TAEF tests where they are needed.

Prompts to review the above recommendations:

• A major update or revision of the above testing frameworks.
• An update on the future of WinAppDriver (or a replacement)
• A need to have more UI tests for Labs experiments.

[mrlacey]

In terms of adding units tests for each experiment.
I'm thinking of adding these:

The red highlight is a shared library containing the actual tests.

The blue highlight is a pair of apps that host the unit tests and allows each test to be run in the context of both a UWP app and a WinAppSDK one. Each app has a link to the shared project (highlighted in Red).

Open questions: (I'm thinking more about these too...)

1. Is this the best place within the solution structure to put the Shared library?
2. Does the experiment specific solution (e.g. CanvasLayout.sln) need host apps to run the tests too?
3. Could experiment specific test host apps be included in the "All" solution and do away with the need for host apps that hold all the tests?

[michael-hawker]

Thanks @mrlacey for the reports. Agree, let's think about keeping things focused on Unit Tests for now and evaluate UI Tests again after we get Labs up and running. At least getting unit tests up and going means we'll learn how things work individually for each experiment and in aggregate for them all.

For your open questions:

2. Yes, each experiment should have the ability to run the tests for its component. Ideally, with either head. We can just stick the shared test project in the main solution tree and then the two heads to run them in the Platforms folder?
3. We just need a way to execute all the tests in the CI, I'm not as worried about needing to run them all locally within VS. If it's easiest to just aggregate them under a 'folder' in the Toolkit.Labs.All.sln than that seems like a fine approach. Otherwise, we could also just look into a special test project head in the all solution which uses wild card source linking or something. - The main requirement here will just be being able to validate the tests from all experiments pass in the CI.

For 1, I'm thinking we may want to quickly sync on how we want the all-up solution to be organized.

A) Experiment - Should we have a folder per experiment, and then have Library, Samples, Tests. And then the folder with the all-up heads?
B) Function - Do we want the library folder, the sample folder, the test folder, and have the projects in there.

We kind of get the existing A split from the folders and individual solutions we've been setting up. Not sure if @Arlodotexe leaned towards the current B structure due to that or if it was just arbitrary at the moment.

I'm not sure how often we'll use the All-Up solution outside of CI and outputting a complete sample app, so not sure which structure is going to be useful at the moment. Any thoughts?

[Arlodotexe]

To keep things simple for the end user, I'm thinking everything they'll be editing for their Lab experiment should appear in the same folder.

It might look like this:

• Solution 'Toolkit.Labs.All'
  • Labs
    • CanvasLayout
      • CommunityToolkit.Labs.WinUI.CanvasLayout
      • CanvasLayout.Sample
      • CanvasLayout.Tests (Shared project)
      • CanvasLayout.Docs
      • Platforms
        • CanvasLayout.Uwp
        • CanvasLayout.Wasm
        • CanvasLayout.WinAppSdk

Of course the actual location of projects can be different on disk. Thoughts?

[mrlacey]

Thoughts?

I thought the intention was that the "All" solution was not where people working on new experiments would be working.
Rather they would be working in the, much smaller, experiment-specific solution.

If this is still the case, we could actually make the number of experiment-specific files in the "All" solution much smaller. I think the only one we have to have in there is the one with the actual new item logic (e.g. CommunityToolkit.Labs.WinUI.CanvasLayout) This would mean not including the projects under the Platforms subdir.

Doing this would really emphasize that work on individual experiments should be done in the experiment-specific solution and the "All" project is for CI and when wanting to see all the experiment samples together.

As more experiments are added the "All" solution risks getting very large when it includes lots of files for each experiment. In turn this will make it slower to work with.

[michael-hawker]

Agree with @mrlacey's thoughts here. All is really about running the samples all-up for exploration and testing vs. specific work (outside of Codespaces that is, maybe we have to see if there's a way later to load a specific experiment in a codespace in the future).

I mean I think the projects need to be part of All as they all get built into the sample app, but we don't have to make it easy to work with them.

It sounds like we're basically leaning towards separating out by Function for All and having as our top-level:

• Libraries
• Samples
• Tests
• Shared Core
• Platform Heads

[mrlacey]

@michael-hawker to clarify, in this proposal for All, does the "Samples" folder need to contain the sample heads for each experiment, or just the shared project?

Only the shared project is needed to run tests in the CI and create a sample app with all the experiments.
Adding all the extra heads makes the build time much longer as it needs to do the release builds of all the heads.

[michael-hawker]

@mrlacey good point. We don't need the heads for each experiment samples/tests, just the shared 'samples'/tests projects. So that'll clean things up a bit.

Because yeah, if they want to run only the experiment, they should switch to that experiment's solution.

[mrlacey]

I'm not aware of whether to keep the open (possibly for a long time) or create separate follow-up issues to track running:

• unit tests in anything other than a UWP & WinAppSDK app container
• Adding UI tests

thoughts? @michael-hawker

[mrlacey]

I think #72, #86, #124 & #160 cover everything not yet implemented.

Are we good to close this?