-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
All-up Testing Goals/Priorities #48
Comments
Research progress to date: For unit tests
UI Tests
Points for further discussion/clarification:
|
Thanks for the update @mrlacey! Lots of good info. @Arlodotexe saw that James posted about their new Uno testing solution here: https://twitter.com/jamesmcroft/status/1508042090013462529?t=_c6BR9vH20ag12T5Ee7mRw&s=19 Do you think it'd be worth taking a look at that to compare to the Uno.UITest solution? And then we can discuss which approaches make the most sense based on our original requirements? |
Here's what my investigations regarding UI testing have identified. There are two options for free cross-platform testing tools that support Windows, Web, & Android.
Both options offer serious challenges to consideration and use. Challenges to using Uno.UITest
Challenges to using Legerity.Uno
Recommendations:
Prompts to review the above recommendations:
|
In terms of adding units tests for each experiment. The red highlight is a shared library containing the actual tests. The blue highlight is a pair of apps that host the unit tests and allows each test to be run in the context of both a UWP app and a WinAppSDK one. Each app has a link to the shared project (highlighted in Red). Open questions: (I'm thinking more about these too...)
|
Thanks @mrlacey for the reports. Agree, let's think about keeping things focused on Unit Tests for now and evaluate UI Tests again after we get Labs up and running. At least getting unit tests up and going means we'll learn how things work individually for each experiment and in aggregate for them all. For your open questions:
For 1, I'm thinking we may want to quickly sync on how we want the all-up solution to be organized. A) Experiment - Should we have a folder per experiment, and then have Library, Samples, Tests. And then the folder with the all-up heads? We kind of get the existing A split from the folders and individual solutions we've been setting up. Not sure if @Arlodotexe leaned towards the current B structure due to that or if it was just arbitrary at the moment. I'm not sure how often we'll use the All-Up solution outside of CI and outputting a complete sample app, so not sure which structure is going to be useful at the moment. Any thoughts? |
To keep things simple for the end user, I'm thinking everything they'll be editing for their Lab experiment should appear in the same folder. It might look like this:
Of course the actual location of projects can be different on disk. Thoughts? |
I thought the intention was that the "All" solution was not where people working on new experiments would be working. If this is still the case, we could actually make the number of experiment-specific files in the "All" solution much smaller. Doing this would really emphasize that work on individual experiments should be done in the experiment-specific solution and the "All" project is for CI and when wanting to see all the experiment samples together. As more experiments are added the "All" solution risks getting very large when it includes lots of files for each experiment. In turn this will make it slower to work with. |
Agree with @mrlacey's thoughts here. I mean I think the projects need to be part of It sounds like we're basically leaning towards separating out by Function for
|
@michael-hawker to clarify, in this proposal for Only the shared project is needed to run tests in the CI and create a sample app with all the experiments. |
@mrlacey good point. We don't need the heads for each experiment samples/tests, just the shared 'samples'/tests projects. So that'll clean things up a bit. Because yeah, if they want to run only the experiment, they should switch to that experiment's solution. |
* Add unit tests For #48 * Update Tests/CommunityToolkit.Labs.UnitTests.Uwp/CommunityToolkit.Labs.UnitTests.Uwp.csproj Co-authored-by: Michael Hawker MSFT (XAML Llama) <[email protected]> * Standardize casing of directory names * shorten "Labs.WinUI.CanvasLayout.Tests" to "CanvasLayout.Tests" * Shorten experiment test project name * removed duplicated default Windows app assets * Remove non-essential assets from test projects * Share Asset files in the test host projects * test projects to use common/shared asset files * standardize XAML formatting * Do not set passive flag if not running xstyler passively * remove experiment specific sample heads from "All" solution and remove folder as will only eve have a single item in it. * remove unneeded experiment head project from WASM solution * update test related package dependencies Co-authored-by: Michael Hawker MSFT (XAML Llama) <[email protected]>
I'm not aware of whether to keep the open (possibly for a long time) or create separate follow-up issues to track running:
thoughts? @michael-hawker |
* Add unit tests For CommunityToolkit#48 * Update Tests/CommunityToolkit.Labs.UnitTests.Uwp/CommunityToolkit.Labs.UnitTests.Uwp.csproj Co-authored-by: Michael Hawker MSFT (XAML Llama) <[email protected]> * Standardize casing of directory names * shorten "Labs.WinUI.CanvasLayout.Tests" to "CanvasLayout.Tests" * Shorten experiment test project name * removed duplicated default Windows app assets * Remove non-essential assets from test projects * Share Asset files in the test host projects * test projects to use common/shared asset files * standardize XAML formatting * Do not set passive flag if not running xstyler passively * remove experiment specific sample heads from "All" solution and remove folder as will only eve have a single item in it. * remove unneeded experiment head project from WASM solution * update test related package dependencies Co-authored-by: Michael Hawker MSFT (XAML Llama) <[email protected]>
Related to #3
Testing Goals
We want to make testing approachable and easy for our developers. We've learned from the main Windows Community Toolkit repo that we can get a lot of flexibility in testing our UI features with the Unit Test infrastructure and the VisualUITestBase helper. This has allowed us a lot of depth in exercising UI without much added complexity to a standard Unit Test.
Ideally, we can have a single test (as written) run across one or more target frameworks to help us validate our extra platform targets. UWP and the WinAppSDK are our main priorities, next coming WASM, and then another like Android would probably be the easiest.
UI Testing has been trickier. Our current setup is pretty complex and a bit fragile. It'd be interesting to learn if there are other options out there that may simplify this process.
Unit Tests (Majority of our tests)
UI Tests (~5-15% expected of needs I believe)
Also
Summary
I think if we can at least get to a point where a single unit test, as written, runs across UWP, WinAppSDK, and at least one other platform, and a UI Test that runs on UWP and WinAppSDK. That should cover 95+% of our testing needs/requirements.
As long as adding tests is simple and easy to test locally (and in the CI for validation), then that should be the main thing. We want to lower the barrier to writing tests so we can write more tests to help us maintain code. This enables us or others to better manipulate existing code to fix issues without worry of breaking core functionality when the original code may not be theirs.
The text was updated successfully, but these errors were encountered: