Optimize CollectPlatformGuids and GenerateByteArrayValue API to reduce build time #113
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Incremental build time Signed-off-by: Ashraf Ali S <[email protected]>
Signed-off-by: Ashraf Ali S <[email protected]>
Below are the comments from V3 patch. and this PR will address those. Besides the PCD default value, I think there are still 2 more things need to check. These 2 things could impact the output of the PcdValueInit.exe. 1. Macros definition in BuildOption There would be a case that the Macros are used in Structure Pcd's definition in header file. User can change the structure pcd's struct def via changing the Macro value from command line or dsc file. 2. The header file list defined in DEC The [Includes] section in dec list folder that contains the structure pcd definition header file. There may be case that user changes the folder under [Includes] to change the header file to change the structure pcd struct definition. Signed-off-by: Ashraf Ali S <[email protected]>
mdkinney
pushed a commit
to mdkinney/edk2
that referenced
this pull request
Feb 8, 2024
During the Incremental build GenerateByteArrayValue used to generate the ByteArrayValue even when there is no change in the PCD/VPDs. which is time consuming API based on the number of PCD/VPDs and SKU IDs. The optimization is that GenerateByteArrayValue is used to store the StructuredPcdsData in a JSON file for each of the arch. and during the Incremental build this API will check, if there is any change in the Structured PCD/VPDs then rest of the flow remains the same. if there is no change then it will return the provious build data. Flow: during the 1st build StructuredPcdsData.json is not exists, StructuredPcdsData will be dumped to json file. and it will copy the output.txt as well. Note: as the output.txt are different for different Arch, so it will be stored in the Arch folder. During the Incremental build check if there is any change in Structured PCD/VPD. if there is a change in Structured VPD/PCD then recreate the StructuredPcdsData.json, and rest of the flow remains same. if there is no change in VPD/PCD read the output.txt and return the data Unit Test: Test1: Modified the Structured Pcds default from DEC file. current flow is executing. Test2: Override the default value of the PCD from DEC file. current flow is executing. Test3: Modified/Override the PCD from DSC file. current flow executing Test4: Modified/Override the FDF from DSC file. current flow executing Test5: update the default value from Command Line.current flow executing Test6: Build without change in PCD in DSC, FDF, DEC and Command Line the proposed changes will be executing, and the return data remains the same with and without the changes. Test7: Build with and without modified the include headers of Structured PCDs. if there is any change in those Structured PCD header then current flow will be executed. With these changes it's helping to save around ~2.5min to ~3.5min of Incremental build time in my build environment. Sample PR: tianocore/edk2-basetools#113 Cc: Yuwei Chen <[email protected]> Cc: Rebecca Cran <[email protected]> Cc: Liming Gao <[email protected]> Cc: Bob Feng <[email protected]> Cc: Amy Chan <[email protected]> Cc: Sai Chaganty <[email protected]> Cc: Digant H Solanki <[email protected]> Signed-off-by: Ashraf Ali S <[email protected]> Reviewed-by: Yuwei Chen <[email protected]>
mergify bot
pushed a commit
to tianocore/edk2
that referenced
this pull request
Feb 8, 2024
During the Incremental build GenerateByteArrayValue used to generate the ByteArrayValue even when there is no change in the PCD/VPDs. which is time consuming API based on the number of PCD/VPDs and SKU IDs. The optimization is that GenerateByteArrayValue is used to store the StructuredPcdsData in a JSON file for each of the arch. and during the Incremental build this API will check, if there is any change in the Structured PCD/VPDs then rest of the flow remains the same. if there is no change then it will return the provious build data. Flow: during the 1st build StructuredPcdsData.json is not exists, StructuredPcdsData will be dumped to json file. and it will copy the output.txt as well. Note: as the output.txt are different for different Arch, so it will be stored in the Arch folder. During the Incremental build check if there is any change in Structured PCD/VPD. if there is a change in Structured VPD/PCD then recreate the StructuredPcdsData.json, and rest of the flow remains same. if there is no change in VPD/PCD read the output.txt and return the data Unit Test: Test1: Modified the Structured Pcds default from DEC file. current flow is executing. Test2: Override the default value of the PCD from DEC file. current flow is executing. Test3: Modified/Override the PCD from DSC file. current flow executing Test4: Modified/Override the FDF from DSC file. current flow executing Test5: update the default value from Command Line.current flow executing Test6: Build without change in PCD in DSC, FDF, DEC and Command Line the proposed changes will be executing, and the return data remains the same with and without the changes. Test7: Build with and without modified the include headers of Structured PCDs. if there is any change in those Structured PCD header then current flow will be executed. With these changes it's helping to save around ~2.5min to ~3.5min of Incremental build time in my build environment. Sample PR: tianocore/edk2-basetools#113 Cc: Yuwei Chen <[email protected]> Cc: Rebecca Cran <[email protected]> Cc: Liming Gao <[email protected]> Cc: Bob Feng <[email protected]> Cc: Amy Chan <[email protected]> Cc: Sai Chaganty <[email protected]> Cc: Digant H Solanki <[email protected]> Signed-off-by: Ashraf Ali S <[email protected]> Reviewed-by: Yuwei Chen <[email protected]>
Wenxing-hou
pushed a commit
to Wenxing-hou/edk2-staging
that referenced
this pull request
Mar 13, 2024
During the Incremental build GenerateByteArrayValue used to generate the ByteArrayValue even when there is no change in the PCD/VPDs. which is time consuming API based on the number of PCD/VPDs and SKU IDs. The optimization is that GenerateByteArrayValue is used to store the StructuredPcdsData in a JSON file for each of the arch. and during the Incremental build this API will check, if there is any change in the Structured PCD/VPDs then rest of the flow remains the same. if there is no change then it will return the provious build data. Flow: during the 1st build StructuredPcdsData.json is not exists, StructuredPcdsData will be dumped to json file. and it will copy the output.txt as well. Note: as the output.txt are different for different Arch, so it will be stored in the Arch folder. During the Incremental build check if there is any change in Structured PCD/VPD. if there is a change in Structured VPD/PCD then recreate the StructuredPcdsData.json, and rest of the flow remains same. if there is no change in VPD/PCD read the output.txt and return the data Unit Test: Test1: Modified the Structured Pcds default from DEC file. current flow is executing. Test2: Override the default value of the PCD from DEC file. current flow is executing. Test3: Modified/Override the PCD from DSC file. current flow executing Test4: Modified/Override the FDF from DSC file. current flow executing Test5: update the default value from Command Line.current flow executing Test6: Build without change in PCD in DSC, FDF, DEC and Command Line the proposed changes will be executing, and the return data remains the same with and without the changes. Test7: Build with and without modified the include headers of Structured PCDs. if there is any change in those Structured PCD header then current flow will be executed. With these changes it's helping to save around ~2.5min to ~3.5min of Incremental build time in my build environment. Sample PR: tianocore/edk2-basetools#113 Cc: Yuwei Chen <[email protected]> Cc: Rebecca Cran <[email protected]> Cc: Liming Gao <[email protected]> Cc: Bob Feng <[email protected]> Cc: Amy Chan <[email protected]> Cc: Sai Chaganty <[email protected]> Cc: Digant H Solanki <[email protected]> Signed-off-by: Ashraf Ali S <[email protected]> Reviewed-by: Yuwei Chen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Incremental build time