crypto/x509: "certificate is not standards compliant" on MacOS #51991

dims · 2022-03-28T19:12:49Z

We hit an error with a unit test we had in Kubernetes and started looking at the impact on end users of kubernetes if the problem is not resolved by the time kubernetes 1.24 is released. More context: please see Kubernetes issue - kubernetes/kubernetes#108956

What version of Go are you using (`go version`)?

$ go version
go version go1.18 darwin/arm64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/dims/Library/Caches/go-build"
GOENV="/Users/dims/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/dims/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/dims/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.18/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.18/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/qw/pkzvlrfs7rn7h6r1x7r57_rw0000gn/T/go-build1513460199=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Please see https://go.dev/play/p/w4rr43vQv7d

What did you expect to see?

success

What did you see instead?

error: x509: “no-sct.badssl.com” certificate is not standards compliant

The text was updated successfully, but these errors were encountered:

dims · 2022-03-28T19:15:56Z

The error seems be introduced here: feb024f#diff-9e2a37df9605e8b207365b51999e6b14e1f5db72b27ad33514dbac502d477c25R212

@liggitt summarized the ask here : kubernetes/kubernetes#108956 (comment) and kubernetes/kubernetes#108956 (comment) on what impact this will have on kubernetes users for whom there was no issues before we switched to go1.17 when they try the kubernetes kubectl command built using go1.18.

Worst case we would like to document scenarios under which users will hit the certificate is not standards compliant error that they were not hitting before.

thanks!

seankhliao · 2022-03-28T20:49:13Z

cc @golang/security

FiloSottile · 2022-03-28T20:54:03Z

Can you share the actual affected certificate? You are unlikely to be hitting the same root cause as no-sct.badssl.com in a unit test, because SCT rules are only enforced for WebPKI certificates.

liggitt · 2022-03-28T20:59:17Z

We only noticed the issue in a unit test checking a cert we expected to be considered invalid (it was a negative test of a cert not signed by a trusted root), so at first we thought we just needed to make the unit test more tolerant of various error messages.

In spelunking around the change in the message, we also ran across https://groups.google.com/g/golang-nuts/c/RGghq2gTWss/m/7GsudTfCAgAJ which indicated requests that previously succeeded were now failing.

Before papering over the change in our unit test by tolerating more validation error messages, I wanted to understand more about which certificates the go validator considers valid that the macOS validator does not. (kubernetes/kubernetes#108956 (comment))

rolandshoemaker · 2022-03-28T23:29:46Z

Apple enforces their SCT requirements on all publicly trusted certificates as part of its base TLS policy (which we use via SecPolicyCreateSSL, since we are generally targeting the web PKI.) Publicly trusted certificates that lack embedded SCTs are very rare, making up something like 0.01% of all publicly trusted certs, but they are out there (the AWS example being probably the most common.)

This will only affect users who are using the bare system certificate pool and are validating certificates which totally lack embedded SCT, with the server providing them via a TLS extension.

It is noted in a TODO in crypto/x509/root_darwin.go that we may want to support passing SCTs passed this way, since we have no way of telling Apple to disable this particular policy (I would need to double check, but it's possible these requirements are not enforced if you use SecPolicyCreateBasicX509, but that would likely also disabled all of the other web PKI policies that we want applied), but how to do that is rather nuanced (since we'd only do anything with them on macOS.)

rolandshoemaker · 2022-03-28T23:49:57Z

Slight side note: the AWS case is a weird one, because I don't think they are sending SCTs at all, despite using publicly trusted certs, so even implementations that know how to pipe SCTs passed via TLS extensions wouldn't work on macOS 🤷.

FiloSottile · 2022-03-29T10:53:57Z

I don't think the SCT policy explains the new error in kubernetes/kubernetes#108956 though, because that's not a publicly trusted certificate. If you want to extract that certificate and share it with us, we can tell you why that one is failing, too.

kubernetes/kubernetes#108956 (comment) papering over the change in our unit test by tolerating more validation error messages, I wanted to understand more about which certificates the go validator considers valid that the macOS validator does not.

We can't really answer this exhaustively, because the macOS verifier has a number of evolving policies that change between OS versions. Note that the platform verifier is only used when the system roots are involved, so behaving like the system is what's expected. I assume k8s clusters use private CAs configured through config.RootCAs for most purposes, which would be unaffected by this.

FiloSottile · 2022-03-29T11:00:21Z

Slight side note: the AWS case is a weird one, because I don't think they are sending SCTs at all, despite using publicly trusted certs, so even implementations that know how to pipe SCTs passed via TLS extensions wouldn't work on macOS 🤷.

Customers should probably reach out to AWS about this. As a short term workaround, it should be possible to add the Amazon root CAs to a x509.SystemCertPool() and use it as config.RootCAs so that the Go verifier is used as well as the system one. (Don't start with an empty pool so that if the root changes you have a chance at not breaking.)

liggitt · 2022-03-29T12:38:46Z

It looks like this change means we no longer get typed TLS errors (e.g. x509.UnknownAuthorityError) when validating using system roots.

That means that special handling of those errors (logging or other fallback paths) that previously worked no longer works in go1.18.

edit: I'll open a separate issue for that, since that is distinct from the "certificate considered valid in go1.17" → "certificate considered invalid in go1.18" issue

liggitt · 2022-03-29T13:07:52Z

opened #52010 for the untyped error issue

liggitt · 2022-03-29T13:10:17Z

I assume k8s clusters use private CAs configured through config.RootCAs for most purposes, which would be unaffected by this.

I also expect that to be true in most scenarios (and in scenarios where it isn't for the certs issued by public CAs to be compatible with system roots, though the referenced AWS issue is evidence that not all certs issued by public CAs are valid).

For k8s' use, I don't think this issue is very significant.

rolandshoemaker · 2022-03-29T15:13:31Z

Oh, that particular test certificate (in TestTLSConfig) is non-compliant in a handful of ways. It's self-signed, but isCA is false, it is missing the cert sign key usage, and it's validity period is likely too long (although I'm not sure if macOS enforces that for self-signed certs.)

calvinbui · 2022-03-30T23:24:39Z

We've had the same issue with connecting to AWS Elasticache Redis servers. Amazon will not support SCTs to avoid publishing customer cluster names in a public log. The connection previously worked fine in 1.17.

jimidle · 2022-03-31T02:35:31Z

A little more background about AWS, or at least how we were connecting to the Neptune graph database service. In case it helps anyone.

Because Neptune is a little "light" on security, you can only connect to it through local/private VPC. This isn't very useful for developers, so we have a VPN to a bastion host for a development only instance of Neptune (Neptune does not have any local installation - it is an AWS service only).

It seems that AWS did not feel the need to put any SCTs in to the Neptune cert, thinking it would only see connections from the secured VPC, and so our connections (via go, it is fine from Java for instance) will fail.

We have raised a ticket with AWS about this. There isn't much can be done about that in Go.

As this is a developer only connection, we have created a reverse proxy with a local CA root. This allows the connection for developers. Hokey, but does what we want for a developer connection. The real solution is of course for AWS to re-issue their certificate, however they say they don't want SCTs in order to avoid placing customer cluster names in a public log (see #51991 (comment) )

bcmills · 2022-04-20T15:24:21Z

This is showing up on the darwin-amd64-10_15 builder as well, though curiously not on any of the other darwin builders.

Marking as release-blocker for Go 1.19 because darwin/amd64 is a first class port.

greplogs --dashboard -md -l -e 'certificate is not standards compliant'

rolandshoemaker · 2022-04-20T16:51:56Z

As far as I can tell this seems, possibly, (this is unbearably painful to diagnose) to be an issue with 10.15.1, which is what the the darwin-amd64-10_15 builder is running. I suspect that updating the builder to use 10.15.6 would fix this, but I have absolutely no clue how viable that is.

bcmills · 2022-04-20T17:32:50Z

I suspect that updating the builder to use 10.15.6 would fix this, but I have absolutely no clue how viable that is.

@golang/release, can you weigh in on that? (How hard is the macOS 10.15 image to update?)

heschi · 2022-04-20T18:44:33Z

For amd64 I think it's maybe a day's work, if we're willing to cut over all the builders at once. Rolling it out gradually will be more unpleasant. I haven't read this issue to judge whether it's a good use of time.

rolandshoemaker · 2022-04-21T15:56:26Z

I don't think there is really any other way to address this issue, given how deeply integrated the TLS client is in the toolchain there isn't really any (safe) way of silently handling/skipping these failures. It's not a high frequency flake though (it seems somewhat correlated with when new certificates are issued) so probably not super high priority.

FiloSottile · 2022-04-21T16:38:59Z

(it seems somewhat correlated with when new certificates are issued)

Can the machine reach the internet? That sounds consistent with a bloom filter window miss on the Apple Valid system, which leads to an OCSP connection to the CA. If that fails, I could see it leading to a vague error like this.

jbg · 2022-05-11T04:38:54Z

In some cases certificates may be deliberately excluded from CT logs to avoid publishing a detailed map of internal infrastructure. (In our case, the certs are associated with DNS names that are only resolvable internally, and which resolve to private IPs.)

e.g. AWS ACM allows disabling CT logging for this purpose, which will result in a valid certificate issued by a trusted CA but not listed in CT logs.

When trying to access a service with such a cert from Go (in our case, using a Terraform provider) on developer (darwin_arm64) machines, we get this certificate is not standards compliant error.

Is there any solution other than logging the certs? Is there any knob in the Go TLS client for turning off the check, which the TF provider could provide a config option to turn?

jimidle · 2022-05-11T04:57:35Z

In our case, we were just needing developers to access a development version of Neptune on AWS. We were doing that with a bastion host and ssh. The Neptune cert is set up exactly as you say, to avoid public logs for what is intended to be a system accessed from a VPC in AWS. We therefore wrote a small reverse proxy instead of a simple tunnel. That works for our case as our code runs in the VPC as AWS intend, other than for developers. As I understand it from the comments here, Go 1.19 will try to address this, but I cannot speak for the Go development team of course.

…

On May 11, 2022 at 12:39:06 PM, Jasper ***@***.***> wrote: In some cases certificates may be deliberately excluded from CT logs to avoid publishing a detailed map of internal infrastructure. e.g. AWS ACM allows disabling CT logging for this purpose, which will result in a valid certificate issued by a trusted CA but not listed in CT logs. When trying to access a service with such a cert from Go (in our case, using a Terraform provider) on developer (darwin_arm64) machines, we get this certificate is not standards compliant error. Is there any solution other than logging the certs? Is there any knob in the Go TLS client for turning off the check, which the TF provider could provide a config option to turn? — Reply to this email directly, view it on GitHub <#51991 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMBSJBB4IPNDNOYUWPLVJM2WVANCNFSM5R4DNYCQ> . You are receiving this because you commented.Message ID: ***@***.***>

sandheepp · 2023-09-15T05:06:43Z

tls: failed to verify certificate: SecPolicyCreateSSL error: 0
I am using aws serverless offline and trying to hit an endpoint, which is also creating issues with Mac M2. This basically makes it very inconvenient for local development after the latest update from apple.

jimidle · 2023-09-15T05:25:06Z

It’s a lien issue because AWS serverless certs deliberately don’t include certain information. You have to connect with certain parameters that exclude that part of verification, but I found that is impossible in certain cases

…

On Fri, Sep 15, 2023 at 13:07 Sandheep P ***@***.***> wrote: tls: failed to verify certificate: SecPolicyCreateSSL error: 0 I am using *aws serverless offline* and trying to hit an endpoint, which is also creating issues with Mac M2. This basically makes it very inconvenient for local development after the latest update from apple. — Reply to this email directly, view it on GitHub <#51991 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMB6LXSHICCFCR3M3K3X2PO7HANCNFSM5R4DNYCQ> . You are receiving this because you commented.Message ID: ***@***.***>

jimidle · 2023-09-15T05:50:16Z

Typo: known

…

On Fri, Sep 15, 2023 at 13:24 Jim Idle ***@***.***> wrote: It’s a lien issue because AWS serverless certs deliberately don’t include certain information. You have to connect with certain parameters that exclude that part of verification, but I found that is impossible in certain cases On Fri, Sep 15, 2023 at 13:07 Sandheep P ***@***.***> wrote: > tls: failed to verify certificate: SecPolicyCreateSSL error: 0 > I am using *aws serverless offline* and trying to hit an endpoint, which > is also creating issues with Mac M2. This basically makes it very > inconvenient for local development after the latest update from apple. > > — > Reply to this email directly, view it on GitHub > <#51991 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAJ7TMB6LXSHICCFCR3M3K3X2PO7HANCNFSM5R4DNYCQ> > . > You are receiving this because you commented.Message ID: > ***@***.***> >

On Mac, the check for certs is more strict and it fails for submariner service. Turning off the check for certs. More info: golang/go#51991 Signed-off-by: Raghavendra Talur <[email protected]>

SaberStrat · 2024-03-08T11:58:17Z

Rebooting didn't help on my M3 Pro either unfortunately. Was still getting that error with our company's self-signed cert.

Went with the workaround of using a linux container with go in it.

Ademord · 2024-04-02T13:53:24Z

Can someone explain maybe also why is this problem happening? I get it when i do "oc login".

jboykin-bread · 2024-04-16T17:28:45Z

We hit this trying to use mirrord with some of our services. For folks trying to work with AWS services that don't use SCTs in their certs locally on macOS, here's a temporary workaround that's working for us.

Download one of AWS's intermediate or Root CAs that issued the problematic cert lacking an SCT. For us to work with ElastiCache, that ended up being this cert: https://www.amazontrust.com/repository/G2-RootCA1.pem
Import the pem file into your macOS keychain, and set the trust policy to "Always Trust" (reference).

The call to SecTrustEvaluateWithError should now pass (https://github.com/golang/go/blob/master/src/crypto/x509/internal/macos/security.go#L200-L215).

On Mac, the check for certs is more strict and it fails for submariner service. Turning off the check for certs. More info: golang/go#51991 Signed-off-by: Raghavendra Talur <[email protected]>

aviramha · 2024-05-20T14:08:01Z

We hit this trying to use mirrord with some of our services. For folks trying to work with AWS services that don't use SCTs in their certs locally on macOS, here's a temporary workaround that's working for us.

Download one of AWS's intermediate or Root CAs that issued the problematic cert lacking an SCT. For us to work with ElastiCache, that ended up being this cert: amazontrust.com/repository/G2-RootCA1.pem

Import the pem file into your macOS keychain, and set the trust policy to "Always Trust" (reference).

The call to SecTrustEvaluateWithError should now pass (master/src/crypto/x509/internal/macos/security.go#L200-L215).

Thanks for reporting this @jboykin-bread ! I just encountered the same issue with another user of mirrord. Would you like to add this case to our FAQ?

jboykin-bread · 2024-05-22T14:14:34Z

We hit this trying to use mirrord with some of our services. For folks trying to work with AWS services that don't use SCTs in their certs locally on macOS, here's a temporary workaround that's working for us.

Download one of AWS's intermediate or Root CAs that issued the problematic cert lacking an SCT. For us to work with ElastiCache, that ended up being this cert: amazontrust.com/repository/G2-RootCA1.pem

Import the pem file into your macOS keychain, and set the trust policy to "Always Trust" (reference).

The call to SecTrustEvaluateWithError should now pass (master/src/crypto/x509/internal/macos/security.go#L200-L215).

Thanks for reporting this @jboykin-bread ! I just encountered the same issue with another user of mirrord. Would you like to add this case to our FAQ?

No worries, I can do that! @aviramha

On Mac, the check for certs is more strict and it fails for submariner service. Turning off the check for certs. More info: golang/go#51991 Signed-off-by: Raghavendra Talur <[email protected]>

dims mentioned this issue Mar 28, 2022

go1.18: tls unit tests don't pass locally kubernetes/kubernetes#108956

Closed

seankhliao added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 28, 2022

liggitt mentioned this issue Mar 29, 2022

crypto/x509: go1.18 stops returning typed errors when using system roots on darwin #52010

Closed

apparentlymart mentioned this issue Apr 1, 2022

Build official releases with Go 1.18 hashicorp/terraform#30768

Merged

bcmills changed the title ~~crypto/x509: "certificate is not standards compliant" on MacOS only with golang 1.18~~ crypto/x509: "certificate is not standards compliant" on MacOS Apr 20, 2022

bcmills added OS-Darwin release-blocker labels Apr 20, 2022

bcmills added this to the Go1.19 milestone Apr 20, 2022

kdrag0n mentioned this issue Nov 24, 2023

buildx unable to pull images from self-signed CA registry orbstack/orbstack#799

Closed

timw mentioned this issue Jan 26, 2024

Server certificate validation on Mac fails when validity period exceeds 398 days riok/Kreya#190

Closed

gopherbot modified the milestones: Go1.22, Go1.23 Feb 6, 2024

ygxbnet mentioned this issue May 18, 2024

mac os 上传报错 ygxbnet/lsky-upload#14

Closed

jacobboykin mentioned this issue Jun 3, 2024

Add cert validation discrepency faq metalbear-co/mirrord.dev#286

Merged

aviramha mentioned this issue Jul 3, 2024

Consider hooking SecTrustEvaluateWithError metalbear-co/mirrord#2569

Closed

seankhliao mentioned this issue Jul 5, 2024

import/path: issue title #68317

Closed

raghavendra-talur mentioned this issue Jul 24, 2024

drenv: Fix submariner for MacOS RamenDR/ramen#1497

Open

gopherbot modified the milestones: Go1.23, Go1.24 Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crypto/x509: "certificate is not standards compliant" on MacOS #51991

crypto/x509: "certificate is not standards compliant" on MacOS #51991

dims commented Mar 28, 2022

dims commented Mar 28, 2022

seankhliao commented Mar 28, 2022

FiloSottile commented Mar 28, 2022

liggitt commented Mar 28, 2022 •

edited

Loading

rolandshoemaker commented Mar 28, 2022

rolandshoemaker commented Mar 28, 2022

FiloSottile commented Mar 29, 2022

FiloSottile commented Mar 29, 2022

liggitt commented Mar 29, 2022 •

edited

Loading

liggitt commented Mar 29, 2022

liggitt commented Mar 29, 2022

rolandshoemaker commented Mar 29, 2022

calvinbui commented Mar 30, 2022

jimidle commented Mar 31, 2022 •

edited

Loading

bcmills commented Apr 20, 2022

rolandshoemaker commented Apr 20, 2022

bcmills commented Apr 20, 2022

heschi commented Apr 20, 2022

rolandshoemaker commented Apr 21, 2022

FiloSottile commented Apr 21, 2022

jbg commented May 11, 2022 •

edited

Loading

jimidle commented May 11, 2022 via email

sandheepp commented Sep 15, 2023

jimidle commented Sep 15, 2023 via email

jimidle commented Sep 15, 2023 via email

SaberStrat commented Mar 8, 2024

Ademord commented Apr 2, 2024

jboykin-bread commented Apr 16, 2024 •

edited

Loading

aviramha commented May 20, 2024 •

edited

Loading

jboykin-bread commented May 22, 2024

crypto/x509: "certificate is not standards compliant" on MacOS #51991

crypto/x509: "certificate is not standards compliant" on MacOS #51991

Comments

dims commented Mar 28, 2022

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

dims commented Mar 28, 2022

seankhliao commented Mar 28, 2022

FiloSottile commented Mar 28, 2022

liggitt commented Mar 28, 2022 • edited Loading

rolandshoemaker commented Mar 28, 2022

rolandshoemaker commented Mar 28, 2022

FiloSottile commented Mar 29, 2022

FiloSottile commented Mar 29, 2022

liggitt commented Mar 29, 2022 • edited Loading

liggitt commented Mar 29, 2022

liggitt commented Mar 29, 2022

rolandshoemaker commented Mar 29, 2022

calvinbui commented Mar 30, 2022

jimidle commented Mar 31, 2022 • edited Loading

bcmills commented Apr 20, 2022

rolandshoemaker commented Apr 20, 2022

bcmills commented Apr 20, 2022

heschi commented Apr 20, 2022

rolandshoemaker commented Apr 21, 2022

FiloSottile commented Apr 21, 2022

jbg commented May 11, 2022 • edited Loading

jimidle commented May 11, 2022 via email

sandheepp commented Sep 15, 2023

jimidle commented Sep 15, 2023 via email

jimidle commented Sep 15, 2023 via email

SaberStrat commented Mar 8, 2024

Ademord commented Apr 2, 2024

jboykin-bread commented Apr 16, 2024 • edited Loading

aviramha commented May 20, 2024 • edited Loading

jboykin-bread commented May 22, 2024

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

liggitt commented Mar 28, 2022 •

edited

Loading

liggitt commented Mar 29, 2022 •

edited

Loading

jimidle commented Mar 31, 2022 •

edited

Loading

jbg commented May 11, 2022 •

edited

Loading

jboykin-bread commented Apr 16, 2024 •

edited

Loading

aviramha commented May 20, 2024 •

edited

Loading