Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HANDSHAKE_SSL_ERROR with RDS and 3.9.3+ #2581

Closed
mattmbrightside opened this issue Apr 9, 2024 · 14 comments · Fixed by #2857
Closed

HANDSHAKE_SSL_ERROR with RDS and 3.9.3+ #2581

mattmbrightside opened this issue Apr 9, 2024 · 14 comments · Fixed by #2857

Comments

@mattmbrightside
Copy link

using mysql2 with AWS nodejs lambdas and AWS RDS.

As of March 2024, our RDS instance was still on the 2019 certificate bundle. Versions of mysql2 <3.9.3 were working fine with calls to createConnection. We updated a project to 3.9.3 and started getting these errors:

{"errorType":"Error","errorMessage":"unable to get local issuer certificate","code":"HANDSHAKE_SSL_ERROR","message":"unable to get local issuer certificate","stack":["Error: unable to get local issuer certificate"," at fn (/private/var/folders/48/8n9w9qzn17s1gpq7tjs3cnv40000gp/T/tmplakzpggq/node_modules/mysql2/promise.js:253:31)

I noticed the reference to #2131 and instructed my team to update the mysql certificate bundle so we could retest. We updated RDS to bundle rds-ca-rsa2048-g1. Trying mysql2 3.9.3 gives the same error.

I noticed the mysql2 issue: #2542 regarding the certificates and waited for the fix and for 3.9.4. We have tried again with 3.9.4 and get the same error on calls to createConnection.

If anyone is successfully using this package 3.9.3 or 3.9.4 with RDS, please let me know what steps are required to get past this error. Thanks!

@mattmbrightside
Copy link
Author

mattmbrightside commented Apr 9, 2024

a bit more info, we are using the AWS RDS Proxy service between the lambda and RDS and our connection options passed to createConnection are

{
  "host": "<rds proxy>.<region>.rds.amazonaws.com",
  "user": "username",
  "password": "password",
  "database": "dbname",
  "ssl": "Amazon RDS",
  "charset": "utf8mb4"
}

@wellwelwel
Copy link
Collaborator

wellwelwel commented Apr 9, 2024

Thanks, @mattmbrightside. That clarifies things 🙋🏻‍♂️
This issue probably can be fixed by joining old and new certificates, but this will create an even larger file.

Unfortunately, even with a RDS in hand, I wouldn't be able to reproduce this error, since it will be created with one of the new certificates.


@sidorares, I think it's important to move this responsibility to a separate dependency soon for more flexibility. Users will soon be alerted to update to version ^3.9.4.

I thought of these possibilities:

A) Create a first release with the old certificates, then create a second release with the new certificates. In this way, the users who need to use old versions could force the version they want from the certificate dependency and we would document it (not yet considering a major because of this) and, by default, the new certificates would be loaded.

B) Just combine all the certificates into a larger file until we think about a major release (this option is independent of a new dependency).

@jeffrey-mutual
Copy link

jeffrey-mutual commented Apr 10, 2024

I would love to help reproduce this error (and create a PR with a fix), but I seem to be having some difficulty doing so.

Amazon RDS
MySQL version 8.0.35
Certificate authority is rds-ca-2019
[email protected]

Config is

{
  host: '<details>.rds.amazonaws.com',
  database: 'db',
  user: 'user',
  password: 'password',
  ssl: 'Amazon RDS'
}

This works fine and I'm able to query the database through mysql2. Anything else you'd like me to check? Could be related to the proxy.

@sidorares
Copy link
Owner

@wellwelwel I created https://github.com/mysqljs/aws-ssl-profiles and added you as a maintainer

Intended use when it's ready would be something along the lines

import awsCaBundle from 'aws-ssl-profiles';

const connection = mysql.createConnection({
  //...
  ssl: awsCaBundle
})l

@dougwilson
Copy link
Collaborator

If only RDS would have just used public certificates like Azure 🤦‍♂️

@wellwelwel
Copy link
Collaborator

wellwelwel commented Apr 10, 2024

Really thanks, @jeffrey-mutual 🤝

When looking into 675dd04, it's possible to see that not all old certificates have been changed, which would explain why only some servers return the authentication error.

@jeffrey-mutual
Copy link

Really thanks, @jeffrey-mutual 🤝

When looking into 675dd04, it's possible to see that not all old certificates have been changed, which would explain why only some servers return the authentication error.

I guess I got lucky then. Both of my databases work on 3.9.4. Without a repro I probably won't work on the PR since I wouldn't easily be able to validate the fix.

@EasyKoen
Copy link

We have the same issue, we also use RDS Proxy.

I found the following information from AWS: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.howitworks.html#rds-proxy-security.tls:

When using a client with --ssl-mode VERIFY_CA or VERIFY_IDENTITY, specify the --ssl-ca option pointing to a CA in .pem format. For the .pem file to use, download all root CA PEMs from Amazon Trust Services and place them into a single .pem file.

https://www.amazontrust.com/repository/ has 5 root CA's, I am going to add them manually to our setup and see if that resolves the problem.

@EasyKoen
Copy link

Well, that actually did the trick.

So there you have it. When you connect to RDS through RDS Proxy, you need the Amazon Root CA's which you can find on the website I posted in the comment above.

@pedrovanzella
Copy link

I can also confirm that adding those certs to the certificate chain fixes the issues for me, using RDS Proxy. Maybe they can just be added to the chain in this repo while the certificates package isn't done yet?

@arturjeber
Copy link

arturjeber commented Apr 28, 2024

Same problem with Azure when using Nextjs 14

const dbConfig = {
  host: process.env.DB_HOST,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  database: process.env.DB_DATABASE,
  ssl: { 
     cert:  process.env.DB_SSL.replace(/\\n/g, '\n')
  },
};
{
  status: 401,
  results: [],
  error: [Error: 00417A49F87F0000:error:0A00010B:SSL routines:tls_validate_record_header:wrong version number:ssl/record/methods/tlsany_meth.c:81:
  ] {
    library: 'SSL routines',
    reason: 'wrong version number',
    code: 'HANDSHAKE_SSL_ERROR',
    fatal: true
  }
}

@robert-pitt-foodhub
Copy link
Contributor

image

We have updated our version from 3.1.1 to 3.9.7 in our SIT environment and this error was introduced across hundreds of lambdas, we are going to revert back to 3.1.1, but thought I would share here to help confirm that the recent changes in 3.9.7 are failing RDS connections

@MarioRomanDono
Copy link

Hi, we have run into the same issue in our team, and we have been able to solve it without directly including the CA certificate. We have two RDS databases, one for our staging environment and one for production (both use rds-ca-rsa2048-g1 as CA). For production, we also use RDS proxy to manage connections. In our case, we are able to successfully connect using TLS and the last version of node-mysql2 (3.10.2) with the following options:

Right now I think this can be a little confusing for anyone intending to use RDS proxy with TLS, because there are no instructions in the documentation for setting a successful connection. Another alternative, as @pedrovanzella said, would be to include these CA certificates to both ssl_profiles.js and aws-ssl-profiles package, and always use ssl: "Amazon RDS". I would be happy to help submitting a PR with the solution of your preference, either updating the documentation or the CA files 😄

@wellwelwel
Copy link
Collaborator

wellwelwel commented Jul 4, 2024

Hi, @MarioRomanDono 🙋🏻‍♂️

The ssl: {} mentioned probably removes the ca property that includes the certificates.

In theory, the issue would be fixed by adding the AWS Proxy certificates to the existing ones in aws-ssl-profiles, then import it to MySQL2.

  • My idea would be to create the possibility of using the proxy certificates individually (namespace export) and, by default, joining the current certificates to the proxy certificates (default export).

Currently the test suite is breaking on some tests after MySQL Server v9, resolving this, certificates is my next priority, but please feel free to contribute first 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants