-
Notifications
You must be signed in to change notification settings - Fork 66
Guidelines for ACVTS OE and Dependency Creation and Management
Most operational environments will generally be of the format "A on B" in which A is an operating system and B is a processor. For example, "Windows 10 Professional (64 bit) on Intel Core i7-4790k (Haswell)".
In these cases, the dependencies should be two in number: one of type "OS" or "Operating System" and one of type "Processor" or "CPU". The dependency fields are free-form, and as such, can be whatever most correctly describes the operational environment in which testing occurred. But these above are the most common.
For example, an operational environment named "Windows 10 Professional (64 bit) on Intel Core i7-4790k (Haswell)" would be defined in the raw JSON as follows:
[
{"acvVersion": "{acvp-version}"},
{
"name": "Windows 10 Professional (64 bit) on Intel Core i7-4790k (Haswell)",
"dependencies": [
{
"type": "Operating System",
"name": "Windows 10 Professional (64 bit)",
"description": "<A short description, if applicable>"
},
{
"type": "Processor",
"name": "Intel Core i7-4790k (Haswell)",
"description": "<A short description, if applicable>"
}
]
}
]Also, please note, if another piece of software runs between the implementation under test and the operating system, perhaps something like OpenSSL or Windows .NET, it is entirely reasonable to continue the "A on B on C" construction as long as needed. For example, "Windows .NET Core 3.0 on Windows 10 Professional (64 bit) on Intel i7-4790k (Haswell)". The above JSON structure would be extended likewise, adding one more dependency of type "Software".
This general concept is how all operational environments should be defined. Always think "A on B". And remember, the implementation under test should never appear in the operational environment. That should always be a separate implementation object.
See the CAVP FAQ Document, section GEN.12, to learn more about what information is required for operational environments.
Dependencies are the lowest level of metadata object and have the greatest level of freedom in their definition. While the fields within a dependency are free-from and can include or exclude whatever a user wishes, there is one important best practice to remember: dependencies should always contain at least a "name" and "type". A "description" is also commonplace but not as critical. Please see JSON sample in the above section for a sample dependency object format.
There are two ways to provide dependency data on an OE update or creation request: by-value or by-reference. These use the fields dependencies and dependencyUrls, respectively.
If, for example, a dependency with ID 3 already exists, and one wishes to add it to the list of dependencies on an object that already has dependencies of IDs 1 and 2, one would use dependencyUrls. This is adding OEs by-reference.
[
{"acvVersion": "{acvp-version}"},
{
"dependencyUrls": [
"/acvp/v1/dependencies/1",
"/acvp/v1/dependencies/2",
"/acvp/v1/dependencies/3"
]
}
]Notice that, consistent with the behavior of update requests throughout all of ACVTS, the update request will set the state of the field provided to exactly that which is provided. this means, the user must provide the full desired list of dependencyUrls, not just the new one. Further information on ACVTS update requests can be found here: Guidelines and Best Practices for ACVP Upate Requests
Also, please note, consistent with standard update request behavior, the name field was not provided in this example, because it is not being updated.
As another example, if a user wished to remove a dependency from the above example, they would similarly provide the dependencyUrls field, omitting the dependency to be removed, as below:
[
{"acvVersion": "{acvp-version}"},
{
"dependencyUrls": [
"/acvp/v1/dependencies/1"
]
}
]If the dependencies field is either already empty, then there is a second option, though one we do not recommend using, which is "dependencies". This allows the user to provide the full body of the dependency being added and it will be automatically created and then added to the OE upon approval. See the example in the "General" heading above for an example of this.
One of the most ocmmon reasons for rejected OE update requests is the issue of orphan dependencies. These are, as the name would suggest, dependencies without an associated parent OE object.
This almost always occurs due to users attempting an OE update to add or remove a given dependency, and using the dependencies field instead of dependencyUrls. This results in the server dropping all current OEs and replacing them with the ones provided. This usually consists of dropping the existing ones, and replacing them with identical copies, plus the addition of the new one, for example.
If an OE already has dependencies, there is almost never a reason to use "dependencies", since it will inevitably lead to orphaned dependencies.
When attempting to add or remove dependencies, please always use dependencyUrls.
If you suspect that you have a reason to use "dependencies", please ask. The only generally-valid time to use it is if the OE has no dependencies at all and they're being added the first time.
Often, there is no traditional operating system between a software and its processor. In these cases, the same logic applies, simply without the operating system. As such, in cases of firmware running directly on hardware, the operational environment contains only one dependency. For example:
[
{"acvVersion": "{acvp-version}"},
{
"name": "ACME CPU 1234",
"dependencies": [
{
"type": "Processor",
"name": "ACME CPU 1234",
"description": "<A short description, if applicable>"
}
]
}
]The CAVP FAQ Document in section GEN.12 outlines the requirements for specifying implementations and operational environments on validations which required testing in a simulator.
The requirement that the implementation be of type "Firmware" still stands and is straightforward on ACVTS.
Regarding the OE, please use the following format for specifying the operational environment when a simulator was used in testing.
{
"name": "Sample Hardware Simulator ABC",
"dependencies": [
{
"name": "Sample Hardware Simulator ABC ",
"type": "Firmware",
"description": "<A short description, if applicable>"
}
]
}In most cases, the OE for hardware implementations should be simply the name of the hardware implementation. The OE would contain simply one dependency, also named the same thing as the implementation, i.e.:
{
"name": "Sample Hardware ABC",
"dependencies": [
{
"name": "Sample Hardware ABC ",
"type": "hardware",
"description": "<A short description, if applicable>"
}
]
}While this may initially seem duplicative, this data format is requested such that the data in the database is useful, specifically the OE data. This is especially important in light of future features which are in development utilizing OE and Dependency data. Providing filler values like "N/A" not only causes data duplication, but provides no useful inputs for these future feature sets.
The CAVP FAQ Document, in section GEN.17, outlines the requirements for specifying the operational environments for validations that involve virtualization and containerization.