Adding NISTDSA API to support ML-DSA-44 and ML-DSA-87

[jakemas]

Issues:

Resolves #CryptoAlg-2725

Description of changes:

This is a sizable PR, I've worked extremely hard to keep the size down in a single PR, but with ML-DSA touching X.509, PKEY, ASN1, there are a lot of changes that need to happen simultaneously to preserve library functionality, so I will document this PR well. Although it appears we touch many files, a lot of them are from renaming/restructuring the code, as well as removing most traces of Dilithium from documentation and replacing it with ML-DSA or more generic alternatives.

1. PKEY structure changes

This PR adds ML-DSA-44 and ML-DSA-87 to AWS-LC. Before this PR AWS-LC only supported ML-DSA-65, as such, we were utilizing the void *ptr of evp_pkey_st, rather than a distinct structure for ML-DSA.

This PR introduces the new structure NISTDSA_KEY * nistdsa_key; inside evp_pkey_st to support ML-DSA and any additional FIPS digital signature algorithms provided by NIST, should AWS-LC wish to include them in the library.

  union {
    void *ptr;
    RSA *rsa;
    DSA *dsa;
    DH *dh;
    EC_KEY *ec;
    KEM_KEY *kem_key;
    NISTDSA_KEY * nistdsa_key;
  } pkey;

While the structure DSA already exists it does not provide support for FIPS-like signature APIs (see more at NIST api conventions.

As, such, NISTDSA_KEY utilizes a new structure to hold public/secret keys, as well as a NISTDSA struct which defines signature algorithm specific information:

struct nistdsa_st {
  const NISTDSA *nistdsa;
  uint8_t *public_key;
  uint8_t *secret_key;
};

typedef struct {
  int nid;
  const uint8_t *oid;
  uint8_t oid_len;
  const char *comment;
  size_t public_key_len;
  size_t secret_key_len;
  size_t signature_len;
  size_t keygen_seed_len;
  size_t sign_seed_len;
  const NISTDSA_METHOD *method;
  const EVP_PKEY_ASN1_METHOD *asn1_method;
} NISTDSA;

This allows us to use a single PKEY structure for all NIST FIPS signature algorithms, much like the existing PKEY struct KEM_KEY *kem_key for KEMS. As a consequence, we are now able to define signature methods such as:

DEFINE_LOCAL_DATA(NISTDSA_METHOD, nist_ml_dsa_65_method) {
  out->keygen = ml_dsa_65_keypair;
  out->sign = ml_dsa_65_sign;
  out->verify = ml_dsa_65_verify;
}

DEFINE_LOCAL_DATA(NISTDSA, nist_dsa_ml_dsa_65) {
  out->nid = NID_MLDSA65;
  out->oid = kOIDMLDSA65;
  out->oid_len = sizeof(kOIDMLDSA65);
  out->comment = "MLDSA65 ";
  out->public_key_len = MLDSA65_PUBLIC_KEY_BYTES;
  out->secret_key_len = MLDSA65_PRIVATE_KEY_BYTES;
  out->signature_len = MLDSA65_SIGNATURE_BYTES;
  out->keygen_seed_len = MLDSA65_KEYGEN_SEED_BYTES;
  out->sign_seed_len = MLDSA65_SIGNATURE_SEED_BYTES;
  out->method = nist_ml_dsa_65_method();
  out->asn1_method = &nistdsa_asn1_meth;
}

much like the existing kem.c file.

These structures will be incredibly useful for subsequent PRs, in which we will be adding de-randomized APIs to support FIPS validation and KATs from seeds. In effect, we will be extending the method to include:

DEFINE_LOCAL_DATA(NISTDSA_METHOD, nist_ml_dsa_65_method) {
  out->keygen = ml_dsa_65_keypair;
  out->keygen_internal = ml_dsa_65_keypair_internal;
  out->sign = ml_dsa_65_sign;
  out->sign_internal = ml_dsa_65_sign_internal;
  out->verify = ml_dsa_65_verify;
}

The directory nistdsa/p_nistdsa.c is used to house generic PKEY operations, as well as NISTDSA specific EVP functions such as EVP_PKEY_nistdsa_set_params, EVP_PKEY_nistdsa_set_params, which work in a similar way to p_kem.c functions EVP_PKEY_CTX_kem_set_params and EVP_PKEY_kem_set_params.

The directory evp_extra/p_nistdsa_asn1.c is used to house generic asn.1 operations for NIST FIPS DSA algorithms.

2. OID changes

We now have real OIDs available from https://csrc.nist.gov/projects/computer-security-objects-register/algorithm-registration. These have been added to the obj files and automatically populated by go run objects.go. We include the following NIDs for ML-DSA:

//2.16.840.1.101.3.4.3.17
static const uint8_t kOIDMLDSA44[]  = {0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x03, 0x11};
//2.16.840.1.101.3.4.3.18
static const uint8_t kOIDMLDSA65[]  = {0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x03, 0x12};
//2.16.840.1.101.3.4.3.19
static const uint8_t kOIDMLDSA87[]  = {0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x03, 0x13};

and a "top-level" NID EVP_PKEY_NISTDSA (similar to EVP_PKEY_KEM to subset all NIST FIPS DSA algorithms. Much like for the KEM API, we include functions SIG_find_dsa_by_nid(int nid) to return actual algorithm specific methods.

3. Testing structure changes

Prior to this PR all testing for ML-DSA-65 was done as a series of g-tests. As we now have the ability to get algorithm/security level specific parameters from the NISTDSA struct, I have overhauled the testing suite to be parameterized over the currently supported signature algorithms:

static const struct ML_DSA parameterSet[] = {
  {"MLDSA44", NID_MLDSA44, 1312, 2560, 2420,  "dilithium/kat/mldsa44.txt", mldsa44kPublicKey, mldsa44kPublicKeySPKI, 1333},
  {"MLDSA65", NID_MLDSA65, 1952, 4032, 3309,  "dilithium/kat/mldsa65.txt", mldsa65kPublicKey, mldsa65kPublicKeySPKI, 1973},
  {"MLDSA87", NID_MLDSA87, 2592, 4896, 4627,  "dilithium/kat/mldsa87.txt", mldsa87kPublicKey, mldsa87kPublicKeySPKI, 2613},
};

For each of the above parameter sets we run:

TEST_P(MLDSAParameterTest, KAT)
TEST_P(MLDSAParameterTest, KeyGen)
TEST_P(MLDSAParameterTest, KeyCmp) 
TEST_P(MLDSAParameterTest, KeySize) 
TEST_P(MLDSAParameterTest, NewKeyFromBytes) 
TEST_P(MLDSAParameterTest, RawFunctions) 
TEST_P(MLDSAParameterTest, SIGOperations) 
TEST_P(MLDSAParameterTest, MarshalParse) 

4. X.509 changes

Changes to X.509 code have been minimized to only the required changes to support this PR. This is already a big PR and I want to avoid expansion wherever possible. For X.509 the changes predominantly are regarding the change of NIDs from the old Dilithium NIDs to the new NIST DSA NIDs/OIDs.

As the OIDs changed, so too much the test certificates kDilithium3Cert, kDilithium3CertNull, and kDilithium3CertParam. These have been regenerated using the following:

Tool used https://github.com/google/der-ascii

1. First generate a valid test certificate in PEM encoding, say cert.pem
2. Convert PEM to DER: openssl x509 -in cert.pem -out cert.der -outform DER
3. Convert DER to ASCII: der2ascii -i cert.der >> cert.txt
4. Make edits to cert.txt as desired, say; certnew.txt
5. Convert ASCII to DER: ascii2der -i certnew.txt >> certnew.der
6. Convert DER to PEM: openssl x509 -in certnew.der -inform DER -out certnew.pem -outform PEM

5. File structure changes

To support the new API, and to maintain code consistency between ML-KEM and ML-DSA and the KEM/DSA API we have moved all ML_DSA specific code to the directory crypto/ml_dsa and all generic NISTDSA code to crypto/nistdsa.

Callouts:

• dilithium/p_dilithium3.c -> nistdsa/p_nistdsa.c all dilithium3 pkey specific functionality is now generic
• dilithium/p_dilithium3_asn1.c -> evp_extra/p_nistdsa_asn1.c all dilithium3 asn1 specific functionality is now generic
• dilithium/p_dilithium_test.cc -> ml_dsa/mldsa_test.cc all dilithium3 testing specific functionality is now generic
• dilithium/sig_dilithium.h -> ml_dsa/ml_dsa.h to match consistency with ml_kem.h
• dilithium/sig_diltihium3.c -> ml_dsa/ml_dsa.c to match consistency with ml_kem.c
• nistdsa/internal.h is a new file, added to maintain consistency with fipsmodule/kem/internal.h

Testing:

See above for description of changes to testing framework.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

[jakemas]

I didn't want to have to remove the enable_dilithium flag in this PR, to prevent it from blowing up in size any further. However, to get the CI to be happy, I think it is inevitable. It's time.

[jakemas]

I'm going to close this PR and open #1949. The reason being, it's much easier to review the diff before making file structure changes. I will make the directory changes after the initial PR is merged.