Fixed multithreaded gradle #1337
Conversation
malice00
force-pushed
the
fix/gradle_multithread
branch
from
3c63aa7
to
98e9a91
Compare
malice00
force-pushed
the
fix/gradle_multithread
branch
from
98e9a91
to
a81e113
Compare
utils.js
Outdated
| @@ -2878,40 +2878,37 @@ export function parseGradleProperties(rawOutput) { | |||
| */ | |||
| export function executeParallelGradleProperties(dir, rootPath, allProjectsStr) { | |||
| const defaultProps = { | |||
| rootProject: subProject, | |||
| rootProject: "root", | |||
There was a problem hiding this comment.
Instead of hardcoding, can we set this to spstr from the for loop in line 2905?
There was a problem hiding this comment.
I'll have look, this was just a quick fix/hack to get things working.
malice00
force-pushed
the
fix/gradle_multithread
branch
from
a81e113
to
9610524
Compare
|
I've renamed some of the variables to make the code less confusing. Some variables where still called 'purl' (or some variant thereof) but where actually a 'bom-ref'. |
| @@ -24,6 +24,7 @@ The following environment variables are available to configure the bom generatio | |||
| | GRADLE_HOME | Specify gradle home | | |||
| | GRADLE_CMD | Set to override gradle command | | |||
| | GRADLE_DEPENDENCY_TASK | By default cdxgen use the task "dependencies" to collect packages. Set to override the task name. | | |||
| | GRADLE_SKIP_MODULES | Comma-separated list of modules to skip during the "dependencies" task. This can be useful if you have modules that would fail the gradle build, eg when they do not have dependencies in the given configuration. Use "root" if the top most module should be skipped, use the name (without leading ":") for all others. | | |||
There was a problem hiding this comment.
Can we add an example to the repo tests?
|
I completely removed the object, since this method should return a String -- even according to its documentation. I assume this was copied from the singlethreaded version of the method, which implements parsing the output as well. Since the multithreaded variant does in fact not parse the output, the |
|
Interesting. Can we add some repo tests to prove that the sbom generated with and without multi threading is identical? We can use this project similar to our snapshot tests. |
|
I'll take a stab at it... |
|
Please let me know once you are done and ready. |
|
Honestly, I'm unsure on how to get a good test in here without setting up an actual (small) project, which would then need Gradle (and therefore Java) and probably some other stuff... Maybe it will be easier to do something like this when you redesign cdxgen in the next major version... |
|
Java and gradle are already setup in repotests, so we only need a sample repo. |
|
Ah, great! I'm not seeing them though -- where are these tests located, so I can check out what we'd need and how to get it running? |
malice00
force-pushed
the
fix/gradle_multithread
branch
2 times, most recently
from
9dc4d67
to
96d06f4
Compare
|
I added a test, but for some reason it's not doing anything... Even with As for that cjd project, I added it and a call to it, but I see no way how this may help in a test -- it simply generates a report which must manually be checked, or am I missing something? |
|
@aryan-rajoria could you kindly take a look and suggest some repo tests? We can merge this PR and add the tests separately if it might be easier. |
|
Does anybody know why we don't see the output of cdxgen in these repotests? Would make stuff a lot easier to check/debug... |
|
@malice00 good points raised. We tried coming up with snapshot tests to track the diff and fail the PR. Testing, in general, is going to take some more effort to get there. As part of another PR, I have temporarily hardcoded "root" as the subProject name. Since there is no response from Ajmal, can we park this PR and resume once the refactoring as part of #1360 is complete? |
Signed-off-by: Roland Asmann <[email protected]>
Signed-off-by: Roland Asmann <[email protected]>
…not return an object! Moved returning the empty String to after logging of information on how to debug errors. Signed-off-by: Roland Asmann <[email protected]>
Signed-off-by: Roland Asmann <[email protected]>
Signed-off-by: Roland Asmann <[email protected]>
Signed-off-by: Roland Asmann <[email protected]>
…alls Signed-off-by: Roland Asmann <[email protected]>
Signed-off-by: Roland Asmann <[email protected]>
Signed-off-by: Roland Asmann <[email protected]>
malice00
force-pushed
the
fix/gradle_multithread
branch
from
6f751ea
to
a0c818e
Compare
|
@setchy are you happy to get this merged and ported to the refactor branch? Will be great if we all could collaborate since freezing the project for a large scale refactor is proving difficult |
|
No need to freeze the project - happy to merging ongoing updates into the refactor branch. I don't want that initial refactor to be more than just moving files around and import updates |
|
@prabhu Awesome, thanks! |
This PR fixes #1335 , although the solution for that issue may need to be redone if someone can reproduce the actual use-case -- instead of a non-existent parameter, a hard-coded String is now used.
After fixing that, I found a several of other problems with the multithreaded solution:
So, the necessary arguments have been multiplied, all arguments have been arranged to their correct places and a new envvar has been introduced to be able to skip problematic modules.