HANDSHAKE_SSL_ERROR with RDS and 3.9.3+

[mattmbrightside]

using mysql2 with AWS nodejs lambdas and AWS RDS.

As of March 2024, our RDS instance was still on the 2019 certificate bundle. Versions of mysql2 <3.9.3 were working fine with calls to createConnection. We updated a project to 3.9.3 and started getting these errors:

{"errorType":"Error","errorMessage":"unable to get local issuer certificate","code":"HANDSHAKE_SSL_ERROR","message":"unable to get local issuer certificate","stack":["Error: unable to get local issuer certificate"," at fn (/private/var/folders/48/8n9w9qzn17s1gpq7tjs3cnv40000gp/T/tmplakzpggq/node_modules/mysql2/promise.js:253:31)

I noticed the reference to #2131 and instructed my team to update the mysql certificate bundle so we could retest. We updated RDS to bundle rds-ca-rsa2048-g1. Trying mysql2 3.9.3 gives the same error.

I noticed the mysql2 issue: #2542 regarding the certificates and waited for the fix and for 3.9.4. We have tried again with 3.9.4 and get the same error on calls to createConnection.

If anyone is successfully using this package 3.9.3 or 3.9.4 with RDS, please let me know what steps are required to get past this error. Thanks!

[mattmbrightside]

a bit more info, we are using the AWS RDS Proxy service between the lambda and RDS and our connection options passed to createConnection are

{
  "host": "<rds proxy>.<region>.rds.amazonaws.com",
  "user": "username",
  "password": "password",
  "database": "dbname",
  "ssl": "Amazon RDS",
  "charset": "utf8mb4"
}

[wellwelwel]

Thanks, @mattmbrightside. That clarifies things 🙋🏻‍♂️
This issue probably can be fixed by joining old and new certificates, but this will create an even larger file.

Unfortunately, even with a RDS in hand, I wouldn't be able to reproduce this error, since it will be created with one of the new certificates.

---

@sidorares, I think it's important to move this responsibility to a separate dependency soon for more flexibility. Users will soon be alerted to update to version ^3.9.4.

I thought of these possibilities:

A) Create a first release with the old certificates, then create a second release with the new certificates. In this way, the users who need to use old versions could force the version they want from the certificate dependency and we would document it (not yet considering a major because of this) and, by default, the new certificates would be loaded.

B) Just combine all the certificates into a larger file until we think about a major release (this option is independent of a new dependency).

[jeffrey-mutual]

I would love to help reproduce this error (and create a PR with a fix), but I seem to be having some difficulty doing so.

Amazon RDS
MySQL version 8.0.35
Certificate authority is rds-ca-2019
[email protected]

Config is

{
  host: '<details>.rds.amazonaws.com',
  database: 'db',
  user: 'user',
  password: 'password',
  ssl: 'Amazon RDS'
}

This works fine and I'm able to query the database through mysql2. Anything else you'd like me to check? Could be related to the proxy.

[sidorares]

@wellwelwel I created https://github.com/mysqljs/aws-ssl-profiles and added you as a maintainer

Intended use when it's ready would be something along the lines

import awsCaBundle from 'aws-ssl-profiles';

const connection = mysql.createConnection({
  //...
  ssl: awsCaBundle
})l

[dougwilson]

If only RDS would have just used public certificates like Azure 🤦‍♂️

[wellwelwel]

Really thanks, @jeffrey-mutual 🤝

When looking into 675dd04, it's possible to see that not all old certificates have been changed, which would explain why only some servers return the authentication error.

[jeffrey-mutual]

Really thanks, @jeffrey-mutual 🤝

When looking into 675dd04, it's possible to see that not all old certificates have been changed, which would explain why only some servers return the authentication error.

I guess I got lucky then. Both of my databases work on 3.9.4. Without a repro I probably won't work on the PR since I wouldn't easily be able to validate the fix.

[EasyKoen]

We have the same issue, we also use RDS Proxy.

I found the following information from AWS: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.howitworks.html#rds-proxy-security.tls:

When using a client with --ssl-mode VERIFY_CA or VERIFY_IDENTITY, specify the --ssl-ca option pointing to a CA in .pem format. For the .pem file to use, download all root CA PEMs from Amazon Trust Services and place them into a single .pem file.

https://www.amazontrust.com/repository/ has 5 root CA's, I am going to add them manually to our setup and see if that resolves the problem.

[EasyKoen]

Well, that actually did the trick.

So there you have it. When you connect to RDS through RDS Proxy, you need the Amazon Root CA's which you can find on the website I posted in the comment above.

[pedrovanzella]

I can also confirm that adding those certs to the certificate chain fixes the issues for me, using RDS Proxy. Maybe they can just be added to the chain in this repo while the certificates package isn't done yet?

[arturjeber]

Same problem with Azure when using Nextjs 14

const dbConfig = {
  host: process.env.DB_HOST,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  database: process.env.DB_DATABASE,
  ssl: { 
     cert:  process.env.DB_SSL.replace(/\\n/g, '\n')
  },
};

{
  status: 401,
  results: [],
  error: [Error: 00417A49F87F0000:error:0A00010B:SSL routines:tls_validate_record_header:wrong version number:ssl/record/methods/tlsany_meth.c:81:
  ] {
    library: 'SSL routines',
    reason: 'wrong version number',
    code: 'HANDSHAKE_SSL_ERROR',
    fatal: true
  }
}

[robert-pitt-foodhub]

We have updated our version from 3.1.1 to 3.9.7 in our SIT environment and this error was introduced across hundreds of lambdas, we are going to revert back to 3.1.1, but thought I would share here to help confirm that the recent changes in 3.9.7 are failing RDS connections

[MarioRomanDono]

Hi, we have run into the same issue in our team, and we have been able to solve it without directly including the CA certificate. We have two RDS databases, one for our staging environment and one for production (both use rds-ca-rsa2048-g1 as CA). For production, we also use RDS proxy to manage connections. In our case, we are able to successfully connect using TLS and the last version of node-mysql2 (3.10.2) with the following options:

• Without using RDS Proxy: ssl: "Amazon RDS". This option includes the CA certificates bundled in

  node-mysql2/lib/constants/ssl_profiles.js

  Line 7 in f23c0f1

  exports['Amazon RDS'] = {

• Using RDS Proxy: ssl: {}. As @EasyKoen posted, RDS proxy connection needs Amazon root CA certificates, but these are not included in ssl_profiles.js (and I believe they are not included in https://github.com/mysqljs/aws-ssl-profiles either). But, if you check the source code of this library, specifically the line where SSL options are used:

  node-mysql2/lib/connection.js

  Line 347 in f23c0f1

  const secureContext = Tls.createSecureContext({

  as well as Node documentation for createSecureContext, if you do not set anything for ca, Mozilla CAs list will be used, which already includes Amazon root CA certificates.

Right now I think this can be a little confusing for anyone intending to use RDS proxy with TLS, because there are no instructions in the documentation for setting a successful connection. Another alternative, as @pedrovanzella said, would be to include these CA certificates to both ssl_profiles.js and aws-ssl-profiles package, and always use ssl: "Amazon RDS". I would be happy to help submitting a PR with the solution of your preference, either updating the documentation or the CA files 😄

[wellwelwel]

Hi, @MarioRomanDono 🙋🏻‍♂️

The ssl: {} mentioned probably removes the ca property that includes the certificates.

In theory, the issue would be fixed by adding the AWS Proxy certificates to the existing ones in aws-ssl-profiles, then import it to MySQL2.

• My idea would be to create the possibility of using the proxy certificates individually (namespace export) and, by default, joining the current certificates to the proxy certificates (default export).

---

Currently the test suite is breaking on some tests after MySQL Server v9, resolving this, certificates is my next priority, but please feel free to contribute first 🚀