-
Notifications
You must be signed in to change notification settings - Fork 272
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhancements for hashed bin delegation #1007
Enhancements for hashed bin delegation #1007
Commits on Mar 26, 2020
-
Test to ensure delegated bin names match prefixes
Add test to ensure delegated bin names are consistent with the hash prefixes that are delegated to the role. This is an implicit assumption of the current implementation, the testing of which will enable us to modify the code with greater confidence. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 790f21f - Browse repository at this point
Copy the full SHA 790f21fView commit details -
Simplify delegated hash bin tests
As we are adding and removing items from the hashed bins and checking for their presence/absence it's simplest if we being with the hashed bins initially empty. If we pass a list of targets when we call delgate_hashed_bins() the delegated roles have an initial set of targets delegated to them, which complicates testing of adding then removing a target to a delegated bin. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for e2407a1 - Browse repository at this point
Copy the full SHA e2407a1View commit details -
Add _get_hash() helper function to repository_tool
This helper will generate a hash of the passed target_filepath using the algorithm defined by the repository_tool's HASH_FUNCTION variable. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for eb01374 - Browse repository at this point
Copy the full SHA eb01374View commit details
Commits on Mar 30, 2020
-
Add _find_bin_for_hash() helper to repository_tool
Add a helper function to determine the name of a bin that a hashed targetfile will be delegated to. Based sketches by Lukas Puehringer in issues theupdateframework#994 & theupdateframework#995 Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 40d1dcf - Browse repository at this point
Copy the full SHA 40d1dcfView commit details
Commits on Apr 1, 2020
-
Refactor add_target_to_bin & remove_target_from_bin
Vastly simplify the implementation, using the _get_hash() and _find_bin_for_hash() helpers added in earlier commits. Furthermore, enable passing of the custom parameter to add_target_to_bin() to better match add_target() Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for ab788e3 - Browse repository at this point
Copy the full SHA ab788e3View commit details -
Simplify the delegate_hashed_bins function based on the semi-pseudo-code by Lukas Puehringer in theupdateframework#995. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 78e2edf - Browse repository at this point
Copy the full SHA 78e2edfView commit details -
Hash target file path's as they appear in metadata
In delegate_to_hashed_bins() duplicate the code which determines the target's file path for writing to the metadata in repository_lib's generate_targets_metadata to ensure that the target path hashed by delegate_hashed_bins() matches the target path in the metadata file. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 642639e - Browse repository at this point
Copy the full SHA 642639eView commit details -
Do not check target file exists in add_target[s]()
The file isn't strictly needed on-disk at the time add_target() and add_targets() are called and this duplicates the check for the file's presence in write[_all]() By removing this check we allow extra versatility in adding targets. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 095fd40 - Browse repository at this point
Copy the full SHA 095fd40View commit details -
Fix replacing targets in add_targets()
Match the pattern in add_target() where if the filepath already exists in roleinfo['paths'] it is updated to replace the existing entry with the new fileinfo. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 468a380 - Browse repository at this point
Copy the full SHA 468a380View commit details -
Enable passing a fileinfo to add_target[_to_bin]()
Add an additional optional parameter to add_target() and add_target_to_bin() which is a fileinfo object matching tuf.formats.FILEINFO_OBJECT This parameter and the custom parameter are mutually exclusive and thus cannot be passed at the same time. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 6c69dae - Browse repository at this point
Copy the full SHA 6c69daeView commit details -
Allow writing metadata without access to targets
Previously, at the time of writing targets metadata, we accessed all targets files and ensured they exist on disk before generating a filinfo dict containing hashes of the target file. This change enables targets metadata to be generated using the fileinfo data that is already stored in the roledb. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 68120ba - Browse repository at this point
Copy the full SHA 68120baView commit details -
Only call logger once for hashed bin info
Merge the logger calls reporting information about the hashed bin delegations into a single logger.info() call to ensure the messages will be grouped together even when integrated into a logging system with multiple parallel sources. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 085c548 - Browse repository at this point
Copy the full SHA 085c548View commit details -
Add test for repo without access to target files
Test the newly added functionality to: * add a target to the repository without access to the target file on disk * write targets metadata without access to target files on disk, by using the existing fileinfo data from the roledb Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 0149804 - Browse repository at this point
Copy the full SHA 0149804View commit details -
Test at least one delegated bin has path_hash_prefixes
When testing delegate_hashed_bins to ensure that hash_path_prefixes map to the generated name of the bin, also check to ensure that at least one of the delegations contains one or more path_hash_prefixes. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 694da23 - Browse repository at this point
Copy the full SHA 694da23View commit details -
Remove custom parameter from add_target_to_bin()
We intend to deprecate the custom parameter of add_target() in favour of using the fileinfo parameter with the custom value populated, therefore it does not make sense to _add_ the custom parameter to add_target_to_bin() Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 05830e6 - Browse repository at this point
Copy the full SHA 05830e6View commit details -
Factor out bin numbers calculations into a helper
Factor out code to calculate the prefix length, total number of prefixes and number of prefixes to be allocated to each bin into a helper function _get_bin_numbers() so that we a) do not duplicate logic, b) can document the logic in one place and c) ensure the logic is consistent. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 8609927 - Browse repository at this point
Copy the full SHA 8609927View commit details -
Add a test for add_target_to_bin that adds a fileinfo
Add some additional checks to test_add_target_to_bin to ensure the code to add a target passing a fileinfo is tested. Signed-off-by: Joshua Lock <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 62e4364 - Browse repository at this point
Copy the full SHA 62e4364View commit details