EAR Support #516

fitzthum · 2024-09-27T23:08:58Z

This is the very beginning of this PR. I still need to fix some issues with the underlying crate, add support for verifying tokens in the KBS, and fixup all the tests, examples, and default values.

See commit messages for more info.

Fixes: #353

Progress so far:

fitzthum · 2024-10-04T22:31:09Z

This is ready for review. I tried to make this as simple as possible, but there are a lot of interlocking pieces. Please read the commit messages. They will hopefully explain what is happening and highlight a couple of significant changes (like not flattening the tcb claims anymore, not supporting multiple policies, etc)

We still have some issues to resolve in the underlying crate before we can merge.

Appraisal is not Send veraison/rust-ear#19 - without this we can't create extensions, which we aren't using anyway yet. we could probably go ahead without a fix, but we have to pin the crate to a hash right before they added extensions or it won't compile (and we will need more commits on top of this for the other issues)
JWT has no expiration field veraison/rust-ear#20 - without this the tokens never expire. interestingly guest-components is fine with this. the code works fine, but it's a security problem since the KBS can't verify that the token hasn't expired.
RawValue has no variant for null veraison/rust-ear#21 - without this JSON deserialization of some values into EAR RawValues causes everything to explode. this may even be triggered by the TDX or SGX claims, which would be bad.

Xynnn007

Thanks Tobin. Nice work about code, tests and documents. I am excited to see that the design that leaves the trusted anchor calculation to users.

I have not dived into the code too deeply, but only for a quick catch for the architecture design.

This PR abondons former AS token format. Maybe we could leave the token plugins still. Probably the difficulty is where should OPA lie. My suggestion is to make OPA a common lib for both legacy AS token and EAR.

In this way, the token type could be configured in the launch toml of AS. Such as

[token]
type = "ear"
# other configs for ear

or

[token]
type = "simple"
# other configs for simple

This can be implemented by

#[serde(tag = "type")]
pub enum AttestationTokenConfig {
    Ear(EarConfig),

    Simple(SimpleConfig),
}

pub type AttestationTokenBroker = Arc<dyn AttestationTokenBrokerTrait + Send + Sync>;

impl TryFrom<AttestationTokenConfig> for AttestationTokenBroker {
    type Error = Error;

    fn try_from(value: AttestationTokenConfig) -> Result<Self> {
        match value {
            AttestationTokenConfig::Ear(config) => {
                let ear_broker = Arc::new(EarBroker::new(config)?);
                Ok(ear_broker as _)
            }
            AttestationTokenConfig::Simple(config) => {
                let simple_broker = Arc::new(SimpleBroker::new(config)?);
                Ok(simple_broker as _)
            }
        }
    }
}

This would give enough flexibility to downstream users to define their different token format, although they might not be standard as EAR. But for CoCo, we can support EAR by default.

Cargo.toml

attestation-service/src/lib.rs

jialez0 · 2024-10-08T06:17:17Z

This PR abondons former AS token format. Maybe we could leave the token plugins still. Probably the difficulty is where should OPA lie. My suggestion is to make OPA a common lib for both legacy AS token and EAR.

In this way, the token type could be configured in the launch toml of AS. Such as

[token]
type = "ear"
# other configs for ear

or

type = "simple"
# other configs for simple

This can be implemented by

#[serde(tag = "type")]
pub enum AttestationTokenConfig {
    Ear(EarConfig),

    Simple(SimpleConfig),
}

pub type AttestationTokenBroker = Arc<dyn AttestationTokenBrokerTrait + Send + Sync>;

impl TryFrom<AttestationTokenConfig> for AttestationTokenBroker {
    type Error = Error;

    fn try_from(value: AttestationTokenConfig) -> Result<Self> {
        match value {
            AttestationTokenConfig::Ear(config) => {
                let ear_broker = Arc::new(EarBroker::new(config)?);
                Ok(ear_broker as _)
            }
            AttestationTokenConfig::Simple(config) => {
                let simple_broker = Arc::new(SimpleBroker::new(config)?);
                Ok(simple_broker as _)
            }
        }
    }
}

This would give enough flexibility to downstream users to define their different token format, although they might not be standard as EAR. But for CoCo, we can support EAR by default.

I concur that the format for generating tokens should be modular. We can default to supporting EAR, but simultaneously, retain the previous simple AS token mode as a plugin option. This approach will provide ample room for customization to the users of this project.

fitzthum · 2024-10-10T16:38:53Z

@Xynnn007 @jialez0 You are both right that this PR makes our token provisioning less generic. In fact, it's no longer generic at all. There are some reasons for this.

First, with EAR tokens the policy is fundamentally tied to the token. Our CoCo token supports just about any policy claims, but with EAR we need to generate an Appraisal. This means that 1) the policy is specific to the type of token we use. 2) A bunch of the code surrounding the policy is specific to the type of token we are using.

The first thing isn't a huge problem. It's not a great user experience to require totally different policies for different tokens, but at least that is configurable at runtime. The second thing is a bit more of a problem. There is code in both the policy engine and lib.rs that is specific to the type of policy that we are using. For instance, the policy engine parses the claims differently, and lib.rs composes the token differently. Supporting generic tokens would require changing some abstractions and adding complexity.

One option would be to move most of the logic into the token generation code. The policy engine would always process all the claims and return the full results. The results would be provided as-is to the token broker. The token broker would also get the TCB claims and any other pieces of the token. I don't really like broadening the scope of the token broker like this. If we want to keep the token broker abstraction the same, we would probably have to add a bunch of conditional logic to the policy engine and lib.rs, which also doesn't seem like a great idea.

It's possible to keep both tokens, but there would be costs. Keep in mind that we would need to create more tests, more examples, extend the docs, etc. Besides the costs, I actually think there are advantages to supporting fewer things. So far we have tried to make Trustee as generic as possible. In general I think this is good, but currently I think we are in danger of giving users too many ways to configure things without showing them any particular way that actually works end-to-end. I think part of the reason why we have some gaps with policies, the RVPS, etc is that we haven't been prescriptive enough. I think EAR tokens are a good fit because they are supposed to be generic, but they are more opinionated.

Anyway, this change wasn't an accident, but I understand that it is kind of scary. I am open to trying to support both, but my question for you is what is the value of keeping the CoCo token?

fitzthum · 2024-10-10T22:00:03Z

Another way we could potentially compromise is by keeping the CoCo token but populating the policy results with an EAR-like Appraisal. I'm not sure if this really makes sense tho.

Xynnn007 · 2024-10-12T03:43:36Z

Hi @fitzthum

Anyway, this change wasn't an accident, but I understand that it is kind of scary. I am open to trying to support both, but my question for you is what is the value of keeping the CoCo token?

Actually we are trying to define some own token format in downstream pre-productions benefits from the original plugin design.

Let's go back to your worries. The core issues are 1. token generation code and 2. user interface.

For 1,
I tried to look into the code about the design of current policy module. Some ideas:

Change the API of policy engine. That takes policy id, data and input as parameters, just as OPA defines. We could have only regorus as backend and abondon the extensibility for policy engine (I have a strong hunch we need to do this because so far there is no indication that we will have a strategic replacement other than rego)
Change the AttestationTokenBroker's issue API to receive parameter reference_data_map, tcb_claims and policy_id. Then the module itself decides how to interact with regorus.
To help KBS recognize the concrete token type, we could add another return value of AS's evaluate function.
For 2,
The current config of AS looks like

{
    "work_dir": "/opt/confidential-containers/attestation-service",
    "policy_engine": "opa",
    "rvps_config": {
	    "remote_addr":"http://rvps:50003"
    },
    "attestation_token_broker": "Simple",
    "attestation_token_config": {
        "duration_min": 5
    }
}

After some change, it would be

{
    "work_dir": "/opt/confidential-containers/attestation-service",
    "policy_engine": "opa",
    "rvps_config": {
	    "remote_addr":"http://rvps:50003"
    },
    "attestation_token_broker": {
        "type": "XXX",
        ...// Other options
    }
    "attestation_token_config": {
        "duration_min": 5
    }
}

The default one we could set as EAR, together with all other e2e examples. Although we provide "too" many ways to run trustee, I believe most users will directly use the default configuration of our e2e test (probably EAR or Simple token), which needs to be ensured to run normally by us. Some advanced users might want do extension, then the value of extensibility values.

fitzthum · 2024-10-14T14:36:25Z

Actually we are trying to define some own token format in downstream pre-productions benefits from the original plugin design.

Ok. This is a valid consideration. Ideally we could converge on one token, but I'm not sure what your requirements are and they might not be public. Keep in mind that the EAR token is in some ways very similar to our CoCo token. For instance, they are both JWT. The EAR token stores a public key in a similar way but with a different path. Consumers can somewhat ignore parsing the AR4SI Trust Claims by just looking at the status field of the Appraisal.

We could have only regorus as backend and abondon the extensibility for policy engine

I think we should try to keep the policy engine modular. My opinion of rego/opa is declining and I would love to provide another option (as long as we don't invent it ourselves).

Btw, this PR also switches from evaluating multiple policies with the AS, to only taking one policy id. If we keep support for the CoCo token, can it just take one policy or does it need to do multiple?

Xynnn007 · 2024-10-15T02:25:09Z

Btw, this PR also switches from evaluating multiple policies with the AS, to only taking one policy id. If we keep support for the CoCo token, can it just take one policy or does it need to do multiple?

One policy is ok for now.

fitzthum · 2024-11-15T20:09:39Z

I will rebase this again next week. All of the underlying issues with the EAR crate have been fixed and updated here. PTAL (at least at the high-level) @Xynnn007 @mkulke @mythi et al

I'd like to either merge this soonish or abandon it and continue working on fixing some issues with RVPS and other stuff that I noticed while working on this.

This commit allows the AS to issue EAR tokens with the help of the rust-ear crate. EAR tokens require particular claims. This creates a binding between the AS policy and the EAR token. Specifically, the policy engine must return an EAR appraisal. The policy engine is still generic. Multiple policy engines could be implemented as long as they create an appraisal. Since policy evaluation is now closely tied to the type of token we are going to generate, make the policy engine more generic and move the logic around calling the policy engine out of lib.rs and into the token broker. There are a few other changes, including that the policy engine no longer takes multiple policies. For now, we only evaluate the first policy in the policy list, but future commits will change this convention so that we only ever think about one policy for the attestation service (until we introduce support for validating multiple devices at once). EAR Tokens also do not use flattened claims. The TCB claims are currently flattened so that we can use the key names as the input to the RVPS. This commit breaks this functionality, but a future commit will change the way the RVPS works to accomodate. There isn't a direct pairing between claim names and reference values, so there is no reason to keep flattening all the claims, especially because the flattening code has some corner cases that it does not support. This commit also adds the init_data_claims and runtime_data_claims to the tcb claims as long as the corresponding claims about the hashes are already there. This will allow the init_data to travel with the token, which will be convenient except if the init_data is too big. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]> Signed-off-by: Xynnn007 <[email protected]>

For EAR tokens we require the public key to be set. There is no option to deserialize a token without validating the signature. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

A keypair is required to sign and validate the attestation token. In the past this was optional, but now it is not. Update the docker-compose manifest and configs to pass in this new keypair and update the docs to tell people how to generate it. This does complicate the user experience, but things are not secure without it. That said, we may be able to implement this automatically in a future PR. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

Now we need to provision a keypair for signing and validatig the attestation tokens. Add this keypair to the docker e2e test Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

The sample attester is enabled by default. Remove setting the environment variable that used to enable it. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

We now require a keypair for signing/validating the attestation token. Add this keypair to our k8s deployment tooling. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

We now require a keypair to sign/validate the attestation token. Add this keypair to the e2e test. Interestingly, we were using a keypair for validating the old CoCo token in this test, but only for the passport mode. Even in background check mode, this keypair is required or the token won't be validated at all. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

Previously we expected the caller of the RVPS to provide a name for the reference value that they wanted. In the AS we were flattening the TCB claims to get this name. Ultimately, the names of the TCB claims do not map directly onto the names of the required reference values. This changes the interface to have the RVPS determine which reference values to send. At the moment, it simply sends all of them. This allows the reference values that are used to mostly be set within the policy itself, which is probably a good idea. In the future, the RVPS should be improved to include a context abtraction that allows groups of reference values to be provided to the AS. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

THe skeleton for a policy that can be used to validate the TCB claims of all platforms in the context of confidential containers. Only sample and snp are supported currently, but this should give a good idea of how to extend the policy to other platforms. There are a few tweaks we can make later, such as supporting `>` or `<` comparisons. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

Update the attestestion service policy docs to describe the requirements for policies that will generate EAR tokens. Also update various example and default policies. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

EAR tokens do not support an expiration claim by default, but fortunately we can use the Extension framework to add an `exp` field that will match what we would expect in a JWT. Add this extension and check it when we validate the token. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

Now that the as policy engine is more generic, re-enable and update the test. This test is slightly tied to EAR. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

The rust-ear crate now supports null json value so we can remove the workaround we had to make sure null value didn't cause an exception. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

Xynnn007

Some more nits and let's see what if a rebase comes

Xynnn007 · 2024-11-18T06:07:33Z

attestation-service/Cargo.toml

 env_logger = { workspace = true, optional = true }
 futures = "0.3.17"
 hex.workspace = true
+jsonwebtoken = "9.3.0"


jsonwebtoken is imported in workspace Cargo.toml. We can directly jsonwebtoken.workspace = true here

Xynnn007 · 2024-11-18T06:07:51Z

attestation-service/Cargo.toml

@@ -59,6 +61,7 @@ serde_variant = "0.1.2"
 sha2.workspace = true
 shadow-rs.workspace = true
 strum.workspace = true
+tempfile = "3.13.0"


Also imported

Suggested change

tempfile = "3.13.0"

tempfile.workspace = true

Xynnn007 · 2024-11-18T06:09:04Z

attestation-service/config.json

-    "attestation_token_broker": "Simple",
-    "attestation_token_config": {
+    "attestation_token_broker": {
+        "type": "Simple",


Do we need to mark this as Ear to align with what we changed in README.md? s.t.

If the policy is not updated, the AS will use the default policy.

Xynnn007 · 2024-11-18T06:09:57Z

attestation-service/docs/policy.md

@@ -1,6 +1,29 @@
 # Attestation Policy (Developing)

 CoCo AS provides a flexible policy support based on Rego to facilitate the customized verification rules.
+The policies are used to general an EAR appraisal, which is part of the EAR token returned by the


We might need to claim that different token type needs different policy formattings. The added part is for Ear

Xynnn007 · 2024-11-18T06:14:03Z

attestation-service/src/bin/restful/mod.rs

-    let policy_ids = if request.policy_ids.is_empty() {
-        info!("no policy specified, use `default`");
-        vec!["default".into()]
-    } else {
-        request.policy_ids
-    };
-


Noticed that https://github.com/confidential-containers/trustee/pull/516/files#diff-a4f0031656042cef2f9c6da58178868fb7f80de2d2f094d33cc38196135e2d1aR199 we will raise an error if no policy is selected. Do we need to keep this part of code?

Xynnn007 · 2024-11-18T06:41:15Z

attestation-service/src/token/simple.rs

+#[derive(Deserialize, Debug, Clone, PartialEq)]
+pub struct TokenSignerConfig {
+    pub key_path: String,
+    pub cert_url: Option<String>,


Please add

#[serde(default = "Option::default")]

Xynnn007 · 2024-11-18T06:41:18Z

attestation-service/src/token/simple.rs

+    pub cert_url: Option<String>,
+
+    // PEM format certificate chain.
+    pub cert_path: Option<String>,


Please add

#[serde(default = "Option::default")]

Xynnn007 · 2024-11-18T06:41:25Z

attestation-service/src/token/simple.rs

+    /// Configuration for signing the token.
+    /// If this is not specified, the token
+    /// will be signed with an ephemeral private key.
+    pub signer: Option<TokenSignerConfig>,


Please add

#[serde(default = "Option::default")]

Xynnn007 · 2024-11-18T06:56:57Z

attestation-service/src/token/ear_broker.rs

+            extensions,
+        };
+
+        let signed_ear = ear.sign_jwt_pem(Algorithm::ES256, &self.private_key_bytes)?;


We now supports to embed jwk part or kid part into the JWT with veraison/rust-ear#29 / https://docs.rs/ear/0.3.0/ear/struct.Ear.html#method.sign_jwt_pem_with_header. Thus we can embed the public key part of self.private_key_bytes. In this way

if KBS side set insecure_key = true. This public jwk can be used to verify the integrity of the jwt.

if KBS side set insecure_key = false. Some extra trusted root certificate should be set on KBS config side and this public jwk can be used to verify the integrity of the jwt

Xynnn007 · 2024-11-18T06:58:35Z

kbs/src/token/mod.rs

+pub enum AttestationTokenVerifierType {
+    Jwk,
+    #[default]
+    Ear,


Please see the previous comment of signing token part of AS' EAR token side. Now EAR token generation will result in a same format of JWT that Simple/ITA uses. Thus we do not the extra token verifier kind but they can share a common one.

mkulke · 2024-11-18T08:34:16Z

kbs/test/config/kbs.toml

@@ -5,7 +5,8 @@ private_key = "./work/https.key"
 certificate = "./work/https.crt"

 [attestation_token_config]
-trusted_certs_paths = ["./work/token-cert.pem"]


I guess we can remove the TOKEN_* code from the Makefile?

mkulke · 2024-11-18T08:45:02Z

attestation-service/src/policy_engine/opa/mod.rs

        let policy_bytes = base64::engine::general_purpose::URL_SAFE_NO_PAD
            .decode(policy)
-            .map_err(RegoError::Base64DecodeFailed)?;
+            .map_err(PolicyError::Base64DecodeFailed)?;


if a library returns an error, i.e. not a generic anyhow, like base64::DecodeError we can just use a #from annotation in the error enum and don't have to map the error explicitly.

mkulke · 2024-11-18T08:56:09Z

attestation-service/src/token/ear_default_policy.rego

+##### TDX TODO
+##### AZ SNP TODO
+##### AZ TDX TODO
+##### SE TODO


will all of those TEE fail to verify if EAR tokens are used?

fitzthum requested a review from a team as a code owner September 27, 2024 23:08

fitzthum marked this pull request as draft September 27, 2024 23:09

fitzthum force-pushed the ear branch 16 times, most recently from a6068df to fe6af93 Compare October 4, 2024 22:21

fitzthum marked this pull request as ready for review October 4, 2024 22:33

fitzthum force-pushed the ear branch 3 times, most recently from 007f110 to e3fa251 Compare October 5, 2024 01:39

Xynnn007 reviewed Oct 8, 2024

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

attestation-service/src/lib.rs Outdated Show resolved Hide resolved

attestation-service/src/lib.rs Outdated Show resolved Hide resolved

fitzthum force-pushed the ear branch 11 times, most recently from f7349c5 to 51654bc Compare November 1, 2024 17:46

fitzthum force-pushed the ear branch from 0d8b3f8 to 4ea9320 Compare November 6, 2024 21:41

fitzthum added 13 commits November 15, 2024 15:15

token: allow KBS to validate EAR tokens

f34f34f

For EAR tokens we require the public key to be set. There is no option to deserialize a token without validating the signature. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

tests: add token keypair to docker e2e test

9806b83

Now we need to provision a keypair for signing and validatig the attestation tokens. Add this keypair to the docker e2e test Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

tests: remove unnecessary env var

ff7f573

The sample attester is enabled by default. Remove setting the environment variable that used to enable it. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

k8s: add token signing keypair to k8s deployments

01c9201

We now require a keypair for signing/validating the attestation token. Add this keypair to our k8s deployment tooling. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

docs: update docs and examples for EAR tokens

4bfaa73

Update the attestestion service policy docs to describe the requirements for policies that will generate EAR tokens. Also update various example and default policies. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

policy: fix as policy test

afbe923

Now that the as policy engine is more generic, re-enable and update the test. This test is slightly tied to EAR. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

ear: remove workaround for null values

13d4931

The rust-ear crate now supports null json value so we can remove the workaround we had to make sure null value didn't cause an exception. Signed-off-by: Tobin Feldman-Fitzthum <[email protected]>

fitzthum force-pushed the ear branch from 4ea9320 to 13d4931 Compare November 15, 2024 21:38

Xynnn007 reviewed Nov 18, 2024

View reviewed changes

mkulke reviewed Nov 18, 2024

View reviewed changes

mkulke added the test_e2e Authorize TEE e2e test run label Nov 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EAR Support #516

EAR Support #516

fitzthum commented Sep 27, 2024 •

edited

Loading

fitzthum commented Oct 4, 2024 •

edited

Loading

Xynnn007 left a comment •

edited

Loading

jialez0 commented Oct 8, 2024

fitzthum commented Oct 10, 2024 •

edited

Loading

fitzthum commented Oct 10, 2024

Xynnn007 commented Oct 12, 2024

fitzthum commented Oct 14, 2024

Xynnn007 commented Oct 15, 2024

fitzthum commented Nov 15, 2024 •

edited

Loading

Xynnn007 left a comment

Xynnn007 Nov 18, 2024

Xynnn007 Nov 18, 2024

Xynnn007 Nov 18, 2024

Xynnn007 Nov 18, 2024

Xynnn007 Nov 18, 2024

Xynnn007 Nov 18, 2024

Xynnn007 Nov 18, 2024

Xynnn007 Nov 18, 2024

Xynnn007 Nov 18, 2024

Xynnn007 Nov 18, 2024

mkulke Nov 18, 2024

mkulke Nov 18, 2024

mkulke Nov 18, 2024

EAR Support #516

Are you sure you want to change the base?

EAR Support #516

Conversation

fitzthum commented Sep 27, 2024 • edited Loading

fitzthum commented Oct 4, 2024 • edited Loading

Xynnn007 left a comment • edited Loading

Choose a reason for hiding this comment

jialez0 commented Oct 8, 2024

fitzthum commented Oct 10, 2024 • edited Loading

fitzthum commented Oct 10, 2024

Xynnn007 commented Oct 12, 2024

fitzthum commented Oct 14, 2024

Xynnn007 commented Oct 15, 2024

fitzthum commented Nov 15, 2024 • edited Loading

Xynnn007 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fitzthum commented Sep 27, 2024 •

edited

Loading

fitzthum commented Oct 4, 2024 •

edited

Loading

Xynnn007 left a comment •

edited

Loading

fitzthum commented Oct 10, 2024 •

edited

Loading

fitzthum commented Nov 15, 2024 •

edited

Loading