Sample projects for integration testing

[Jarod42]

I made some sample projects to test basic features on https://github.com/Jarod42/premake-sample-projects

It has already allowed to find some issues (and some has been fixed :-) ).
Current Github actions are by generator (we don't have matrix badge to fill automatically the result table :-/ )

Do you have suggestions about other features to test?

[starkos]

It looks like the test is considered "passed" if the generated project is able to build successfully; is that correct?

[Jarod42]

I currently considerer OK if all the following are ok:

• generation of project from premake script.
• build from generated solution.
• running app from above build.

I didn't check first step return value.
Second step is checked (but Codelite doesn't return failure build from command line (reported bug))
Last step is checked and so validate previous step too.

I workaround premake4 scripts in case of failure (as missing pch support for Codelite in premake4), as I don't expect bug fix for it.

[Jarod42]

I added more modules to test:

• samples project unmodified (one of your fears about that way of testing).
• spot some issues, most of them fixed.

I really think sample projects are the way to validate regular generators.

[samsinsane]

Honestly, I'm not really sure. I think it can be pretty misleading about what works. For example, the linked repo says that "gmake" works, but we're getting reports that it doesn't work for MinGW. Right off the bat I've lost confidence in these. Similarly, what exactly is being tested under "gmake"? Just Linux? What about BSD? macOS? Windows? Just GCC? What about Clang? MSC? Just C++? What about C? C#? F#? D? Every single "action" has a similar set of questions. Furthermore, these are really only going to be CI tests, these won't be things that people run on their own machines since you can't run some of the tools on all platforms - will people fix the issues they introduce after however long it takes to execute every single project?

I think there's some value in validating that Premake doesn't fall over - like it did for MonoDevelop for several months. I'm just kind of struggling to see the value in the tests that are there. The issues that have been found aren't exactly "bugs" as much as they are "missing functionality". Obviously, huge thanks to @Jarod42 for adding in that missing functionality, however, I'm not seeing the value beyond "Oh yeah, CodeLite support is pretty minimal", which we already knew.

I don't think there's a great deal of harm in having some form of validation, but I'm not sure an endless list of projects is the right path forward. I'm happy for it to be trialled, but as I kind of hint at above, I want to see incredibly specific descriptions of what is actually being tested and marked as working, not just "gmake" or "codelite".

[Jarod42]

Honestly, I'm not really sure. I think it can be pretty misleading about what works.

Not much misleading than current unit-tests. Maybe the way I report the result.

I currently just have 5 sample projects (excluding the empty one), and each generator except msvc/gmake fails those one initially and need patch.

I remove one advanced project as you don't know the syntax you want for define with quote.

Similarly, what exactly is being tested under "gmake"?

Yes my tests are limited. I currently test each action only for one platform (and default toolset) for simplicity, but it can be extended if needed (to the limit of what github action propose ;-) )

I currently don't have indeed sample-projects for toolset or language other than C++

So indeed there are missing projects to test other features/environment.

Every single "action" has a similar set of questions.

On the contrary I would say. with my (rudimentary) matrix, I know if feature is supported.
(it might still have bugs not spotted by tests as any tests. In those case current test should be improve, or other tests added)

these won't be things that people run on their own machines since you can't run some of the tools on all platforms - will people fix the issues they introduce after however long it takes to execute every single project?

Test can be done locally for each installed IDE.
but indeed, remote CI provided the missing piece to test what you cannot do locally.

Currently, running tests on all projects take around 10s. similar time to aptget install the IDE.
Most time spend is to rebuild premake5 (2 minutes on windows, 30/40s on linux).

The issues that have been found aren't exactly "bugs" as much as they are "missing functionality".

I won't call those "missing functionality" as there is an entry in Premake.
"selecting shell" or "default config" are missing functionalities.

I don't think there's a great deal of harm in having some form of validation, but I'm not sure an endless list of projects is the right path forward.

Why endless?
And i a previously comment, those tests can be shared across generators, contrary of current test which were repeated (when present) for each generator.

If I would have to complains about your current UTs, it is not there are missing cases (even if there are),
but the fact that most generator-tests just check template is correctly replaced verbatim;
so changes in implementation mostly require to change test too.

I'm happy for it to be trialled, but as I kind of hint at above, I want to see incredibly specific descriptions of what is actually being tested and marked as working, not just "gmake" or "codelite".

I tried to describe shortly what each project tests.

Way of display is not ideal I agree (current display are already been improved).

In ideal, all case of the matrix would be green. So it should be replaced by a matrix which would just pinpoint what IDE doesn't support natively (as option IDE-specific (mostly for MSVC) which are not supported by other toolset/IDE).

Another important point, IMO, is that, it might be a check/condition to include/remove a module in/from core.
(I'm happy that Codelite is in Core, but other IDEs/build systems seems more important/popular)
There is a question in premake-ninja to know what should be done to have that module in Premake's core.

A last point, it allows to shows usage example of the feature, (even if there are mostly artificial currently).

[samsinsane]

Not much misleading than current unit-tests. Maybe the way I report the result.

That's exactly what I'm saying.

On the contrary I would say. with my (rudimentary) matrix, I know if feature is supported.

You don't. Your matrix says that "MSVC" supports all those features. You're only testing VS2019. How do you know that the code that supports that feature isn't wrapped up in a check for _ACTION == "vs2019"? You don't, because you're only testing VS2019.

I won't call those "missing functionality" as there is an entry in Premake.

That's literally what it is. You're not discovering a bug, like we're outputting a malformed XML element, or we're using the wrong values in an XML element. You're discovering that CodeLite doesn't support something, which we already know from the unit tests because there aren't any tests for those things.

Why endless?

Well...

I currently just have 5 sample projects

"currently"

I currently test each action only for one platform

"currently"

I currently don't have indeed sample-projects

"currently"

In those case current test should be improve, or other tests added

"other tests added"

This is why I say endless, or do you genuinely believe that your 5 tests are all we will ever need? There will always be more tests to add, that's two sets of tests that we're maintaining and extending.

---

Honestly, I'm not really sure.

I'm leaning pretty hard towards "definitely not" after this reply.

[Jarod42]

Your criticisms seems to apply to any unit/integration tests:

• don't cover all features.
• don't cover all cases/paths for each feature.
  (- don't cover all combinations of features interaction).

Which is true, but tests are useful. (It seems to me that you are against any kinds of tests).

Naming can be improved or just "hidden" as your current UTs. It seems we agree on that, but I don't think it is the main point anyway.

About number of tests,
I have 5 projects which are independent of generator (I depend of toolset for msvc (for pch, and for buildoption as I didn't find a common testable buildoption)),

Assuming one UT by project (which is wrong as my project tests more a category that a single feature), and 6 generators,
You would have to write 6*5 tests.

• For each new features added, #generators tests should be added.
• For each new generator, #features tests should be added.

Sample project seems more scalable:

• For new generator, no new tests to write (potentially edit some existing (failing) projects).
• For new feature, add a sample project.

In addition, I think projects are:

• more reliable as I test execution too. whereas UTs just test a fragment of text.
• more consistant (same project for each generator (with potential explicit filter))
• and so more simple: the one who adds new feature should know how to create a project using it.

[samsinsane]

(It seems to me that you are against any kinds of tests).

I'm not. I'm against the way these particular tests are being pushed. Whether you're intending this or not, I'm inferring from your comments that you've found the silver bullet for Premake issues and I just don't see this. So far, I can see that you've found CodeLite was missing support for some APIs, you could have determined this from the unit tests or from just doing a Ctrl+F in the CodeLite generator. We already knew the CodeLite generator was only supporting a subset of APIs, we know this about the XCode generator too. Is that valuable? To me, it's not enough to warrant all this extra cruft in the repo.

I think this discussion is happening way too early in the process. For me, these kinds of tests are to prove that Premake emits things wrong, not that it doesn't support some feature for some generator. Once you start having tests that prove Premake emits things wrong, then I'll be open to the idea of these tests being added but until then this is all premature and doesn't add enough value for me.

[starkos]

Lively discussion ! So…this is just the unit vs. integration test debate. They both have value, they both have strengths and weaknesses. I'm a big proponent of unit testing, and these specific test suites have already proven themselves invaluable time and again. We are keeping them, that is not negotiable, end of discussion there.

What the unit tests don't do a good job at, as @Jarod42's integration test suite has already shown, is proving that a feature works fully end-to-end, for any given toolset. So integration tests can have value for this project. What's up for discussion is scope, feasibility, maintainability…bang for the buck, really.

I do share @samsinsane's concern re: the result matrix, which seems to imply "Visual Studio is 100% correct and working" and we know that isn't the case. Case in point: #1685. We already know that there are lots of other gaps like this and these integration tests, and they stand currently, don't reflect that.

IMHO, to get over the bang-for-the-buck line I'd like these scripts to test and show results at the feature level. For instance, I'd like a way to see that the "static link modifier" feature is covered by these tests, and where it is passing and failing. @Jarod42 has a lot of plumbing for that here, it is just testing and reporting at too broad a level. Again IMHO, but I think it would be much more compelling if it identified a few core features and then provided a test matrix just for those, to start.

Recap:

• IMHO, I do think this effort has value and that is worth continuing, so long as @Jarod42 is up for it
• IMHO, I think there needs to be a clear "here are the actual features being tested" matrix, and the test scripts need to clearly markup where and how each feature is tested (would want a "feature ID" list)

[samsinsane]

I agree with most of what you've said here. My main concerns are:

• The results matrix is misleading about support in the different generators. Versions, Toolsets, Platforms, Architectures, and probably more. There's a lot to test, and interpreting the current matrix as proving anything works for a whole generator is incredibly dishonest.
• What value is being provided over the current unit tests, searching the codebase and/or just manually documenting what works and doesn't work? And does that value outweigh the extra effort of maintaining and extending these new tests?

To be clear, I'm not against ever adding these kinds of tests. I'm also happy with whatever you decide here @starkos.

[Jarod42]

I already asked for features to test.
I didn't even know that the syntax :static was possible. It seems it is an advanced feature (and toolset specific), whereas basic and portable ones are not all tested yet.

Whereas my matrix gives incomplete (I disagree on misleading :) ) feedback, it seems better than no feedback at all where we might expect in the same ways that it should works (your CI are validated after all).

How do you want I display that matrix?
I already list keywords used by project.
(I might split project to use/test one keyword by project, but would it be more misleading for you to have 20 passed tests instead of 5; in addition some keywords work together).

My next step is to fix premake-cmake (4/5 failed tests originally (fixed)).
I don't think I will add other generators (I would like to add compile_commands.json (but not a regular generator) and vscode which seems an interesting target, but https://github.com/Cannedfood/premake-vscode is just WIP, and I don't know that IDE to contribute anyway).

Then I would probably add tests for Custom rules. (Added, and most generators fail).

[starkos]

I already asked for features to test.

Yeah, well, you know, we have a lot going on. There's documentation and source code. 🤷

How do you want I display that matrix?

I haven't given thought to how it should actually be displayed. The point being that instead of saying "Visual Studio works" (which it clearly does not for all cases) I would like to see "Shared library projects work in Visual Studio, Xcode, and Gmake"…with that statement backed up by a documented (code comments) test that actually generates, builds, and validates the results to show that shared library projects really do work.

IF the tests get to that point, then I'd be happy to have another look at this. But as things stand now I'm much happier about your recent core code fixes—thank you again!—than the tests you're using to drive them. IMHO while a full integration test suite would be lovely I very much doubt we have the resources to maintain a truly useful number of tests—but I'll keep an open mind on it.

[Jarod42]

Yeah, well, you know, we have a lot going on. There's documentation and source code.

I meant that you might know which features are most important/useful to test;
or features which are the same (I initially though that rules/custom commands were sugar syntax between both).
I already tried to categorize keywords by features and try to do it in a good order.

The point being that instead of saying "Visual Studio works".

I didn't say that (or it is not my intention in the display and it might be our communication issue).
I just say that msvc 2019 passes my sample projects:
so handles (at least in some cases)

• includedirs, sysincludedirs, forceincludes, defines, buildoptions
• kind ("ConsoleApp", "staticLib", "SharedLib") and links between projects.
• pre-compiled header (pchheader/pchsource)
• prebuildcommands/postbuildcommands
• custom-command with buildcommands/buildinputs/buildoutputs/buildmessage/compilebuildoutputs
• custom-rule with rule, rules, fileExtension.

Which is now also true (except for custom rule (fixes in progress)) for other generators.

So, I might split project-01-include-define-and-buildoptions to have 5 projects instead of 1 (one for includedirs, one for sysincludedirs, one for forceincludes, one for defines and one for buildoptions).
Indeed some generators handle includedirs but not sysincludedirs.
but not sure if it is what you want.

About document my tests/projects, not sure what you want neither.
I try to have minimal and complete premake files. I only put comment when I think it is useful/required (as for (https://github.com/Jarod42/premake-sample-projects/blob/master/project-03-pch/premake5.lua)).
that part is not in the readme.md neither.

[starkos]

You keep saying "not sure if it is what you want"? Umm, this isn't my initiative? You asked the community if they found these tests useful. So far they seem to have said "not so much but maybe if…". What's next is up to you.

Again, appreciate the contributions you're making to core!

[Jarod42]

You keep saying "not sure if it is what you want"? Umm, this isn't my initiative?

:-)
Yes, it is mine.

You asked the community if they found these tests useful. So far they seem to have said "not so much but maybe if…".

I try to feel the gap of the "maybe if ..." to increase the chance to reconsider the current "no":

• Missing resources to update the projects:
  I tried to show that it is not time consuming (and even less than current state)
  (and most time is spend to fix(/add missing features for) generators)
• Incomplete:
  I am adding some projects (even for features I don't use for my projects which use Premake).
• ambiguous results report:
  That is the point that I try to fix from the current conversation.
  It should normally be easily fixable as it is just the form, and not the content, but I have difficulty to see what is wrong for you/the community.
  I may add untested features entries (cppdialect, entrypoint, languages "D", "C#", link with existing libraies, ...), without "unknown" as result.
  Alternatively, I might add the matrix completely...