Missing LE CA?

[ishioni]

When doing backups to an https minio endpoint, i get

 tls: failed to verify certificate: x509: certificate signed by unknown authority

The certificate is a wildcard LE certificate

* Connected to s3.services.movishell.pl (10.1.4.2) port 9000 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=*.services.movishell.pl
*  start date: Jun 29 08:42:23 2023 GMT
*  expire date: Sep 27 08:42:22 2023 GMT
*  subjectAltName: host "s3.services.movishell.pl" matched cert's "*.services.movishell.pl"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/1.1
> GET / HTTP/1.1
> Host: s3.services.movishell.pl:9000
> User-Agent: curl/7.88.1
> Accept: */*

[frezbo]

Could you try with this image: ghcr.io/frezbo/talos-backup:v0.1.0-alpha.0-dirty?

[ishioni]

@frezbo Still the same. I think aws-go-sdk is expecting the ca-certificate file in /etc/ssl/certs/ca-certificates.crt, but ghcr.io/siderolabs/ca-certificates:v1.4.1 is putting it in /etc/ssl/certs/ca-certificates/cacert.pem. At least that's what I think looking at aws/aws-sdk-go#2322, but I'm having trouble replicating your build setup to build my own image to test..

[frezbo]

could you try again, pushed with the fix, same image

[ishioni]

Nope, still same result :( (yeah i checked the image sha to make sure it's the newest one)

[frezbo]

could you try again, put the certs in wrong place: ghcr.io/frezbo/talos-backup:v0.1.0-alpha.0-dirty: digest: sha256:8dd68246055e7e94b218b89464b54695bf1337875986a87d8e288a1884cf94b7

[ishioni]

New errror

2023/08/29 11:57:16 failed to push encrypted snapshot: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "https://s3.services.movishell.pl:9000/talos/backups/talos-2023-08-29T11%3A57%3A12Z.snap.age?x-id=PutObject": tls: failed to verify certificate: x509: failed to load system roots and no roots provided; open /etc/ssl/certs/ca-certificates.crt: permission denied

[frezbo]

hmm that's weird, it's running as root, unless the k8s manifest set it othwerwise

[ishioni]

Nah, in the example cronjob you set uid/gid to 1000
Edit: runs fine as root
Edit2: AH! It runs fine as root in the original image as well. It just couldn't find the ca chain as a user!

[frezbo]

Nah, in the example cronjob you set uid/gid to 1000 Edit: runs fine as root Edit2: AH! It runs fine as root in the original image as well. It just couldn't find the ca chain as a user!

will fix the file permissions

[frezbo]

Could you try this: ghcr.io/frezbo/talos-backup:v0.1.0-alpha.0-dirty: digest: sha256:290d4827daa78fa8839ec171906169a8f0e655ad4ea2efabce7e4242daffcd62

[ishioni]

Nope, now it's back to not seing the CA

[frezbo]

so weird, I'll check on this, thanks

[frezbo]

okay, that was a mistake from my side: ghcr.io/frezbo/talos-backup:v0.1.0-alpha.0-dirty: digest: sha256:9c9536887383564939231b05d3008b9143df69957089c1522a4f53e773a227a6 can you try this?

[ishioni]

Success! That one worked!

[frezbo]

will get this fixed soon