Offer up-to-date test results from real hardware

[PatrickvL]

In order to be able to compare test results when running within an emulator, against test results when running on a hardware Xbox, we need to have a repository that contains up-to-date test results from as many relevant hardware variations as possible.

The process to get these results should be made as easy as possible, so it can be repeated by anyone that has access to the required equipment.

Thorough documentation on all relevant aspects is a must for this!

[PatrickvL]

Test results should be identified by :

• xbox_kernel_test_suite build hash
• Run timestamp
• Submitter
• Name and/or description
• Test environment; Either :
  • Used emulator + version / build hash + relevant settings, or
  • Xbox hardware revision + bios version

[x1nixmzeng]

Setting up a bot would be trivial. It could watch for pull requests (or use the Github API) and upload the XBE published from travis to the Xbox.

When I find the time I plan on setting this up - but in the meantime we need some actual test cases added to the project 🤣

[RadWolfie]

I'm incline to close this ticket since PR reviewers should perform test on their hardware first before approve merge action. Having xbox hardware running 24/7 isn't necessary to verify the test results are passing or not. With that said, there are pretty much no active contributors writing tests at the moment. But I wouldn't consider myself as active contributor anyway.

[PatrickvL]

This issue is more about having access to the output/results of hardware runs of the test-suite, and less about having Xbox hardware running tests 24/7. Having access to these results makes it easier to verify emulator results when no hardware is available

[RadWolfie]

Okay, xkts isn't going to be responsible for emulators' issues. Instead, xkts should be intended to test on hardware directly to ensure all variant of tests are conducted according #4 guideline. The emulator projects can use xkts to verify their work. So, it's up to them to fix their end of issues. Unless bad coding occur from here, then we need to be informed about it.

Imho, this ticket is a bit too high expectation to test on many various hardware. However, we should be testing on specific hardware rather than many various hardware consoles. There are so many kernel APIs for sure will have, almost, exactly same results. Only a few will need different set of hardware consoles. Such as:

• dev kit / debug kit
• retail console
• chihiro (type 1 and 3)

Instead of each specific retail console versions.

In the log file, we can have...

• build hash
• bios version
• and maybe hardware revision if identification is possible.

The following above could be create in separate ticket / pull request to include easier support for someone who may have different revisions of hardware if they wish to do so. The submitter and name/description may also can be done through config file depending on if it's necessary to have. I very much doubt we will have run timestamp logged. Since system time will not be stored after long period of powered off state, aka disconnected from wall outlet.

However, these documentations should be useful for the PR reviewers, aka xkts team, who has working hardware, including pull request creators if they're able to. If the PR reviewer don't have working hardware to verify the tests, then what would be the point of accepting pull requests that aren't tested on hardware?

About the issue of having access to the hardware, there are at least two or three members, including myself, from our team are able to perform the tests.

[RadWolfie]

Here's what I'm thinking about Name field... it may need to be in a separate file to read from. Mainly to make things easier to edit config.txt file once and save log file with name as suffix. Except for the separate file contain the name input cannot print any failure/successful to any sources. Well, maybe only to the DbgPrint function which could be possible to use from.

For example: The file, name.cfg (or name.txt), contain machine1 text which will produce kernel_tests-machine1.log file (otherwise default to usual kernel_tests.log file). Which will help multi-download the log files from different machines. Plus able to edit config.txt once for all machines than re-edit each machine's config.txt file.

Of course there will be Name: <user input> output in the log file as well.

[RadWolfie]

Since this ticket is mainly about requested fields to be output into log file. They have been implemented by pull request #95 and #98. Closing this as completed.

As for community-enabled to supply up to date test results, there has been no activity from this for about 4 years. A better viable solution would be ticket #5 for easier process to verify the changes. Other than that, only 2 or 3 on the team currently have access to the retail hardware.