Skip to content

Login.gov SAML certificate rotation steps

Jump to bottom
Fuhu Xia edited this page Mar 22, 2024 · 13 revisions

The login.gov SAML certificate needs to rotated every year by updating the IdP metadata endpoints. It happens in the spring time. Once we are notified, or it is announced at https://developers.login.gov/saml/#annual-certificate-rotation, we have one month's time to get it done. We will need to update the IdP metadata for all applications that use the login.gov as IdP, including its sandbox (https://idp.int.identitysandbox.gov/) and production (https://secure.login.gov/) environments. At the same time, optionally, we will do the annual SP certificate renewals for applications in production environment.

Process overview

  1. IdP metadata update is done in application code. Get the latest metadata and update the offline copy of the IdP XML and have the code deployed. No change needed on login.gov dashboard.
  2. [Optional] New SP certificate is generated via command line. Its private key needs to be base64 encoded and added to cloud.gov CF environment variables. Public certificate needs to be updated on login.gov dashboard. The CF update and dashboard update should be synchronized to minimize downtime. For apps on login.gov sandbox dashboard, we can update them ourselves. For apps on login.gov production, we need to update a mirror copy of production app on sandbox dashboard, verify the change works, then submit a change request to login.gov support. The request needs to be submitted at least one week prior to scheduled deployment date. Login.gov deployment schedule is weekly on Thursday.

Steps to update IdP and SP certificates in different environments:

Update IdP metadata:

Download lastest IdP metadata, for example, for the year of 2021 metadata rotation, the endpoints are https://idp.int.identitysandbox.gov/api/saml/metadata2021 for login.gov sandbox IdP, or https://secure.login.gov/api/saml/metadata2021 for login proudction IdP. Replace the IdP metadata XML for each app at:

  • Catalog (local docker) uses ckan/saml2/idp.xml

  • Catalog (developmenet) uses ckan/setup/login.sandbox.idp.xml

  • Catalog (staging) uses ckan/setup/login.production.idp.xml

  • Catalog uses ckan/setup/login.production.idp.xml

  • Inventory (developmenet) uses config/login.sandbox.idp.xml

  • Inventory (staging) uses config/login.production.idp.xml

  • Inventory uses config/login.production.idp.xml

Renew CKAN app SP certificates [Optional]:

Renew CKAN app SP certificates for 1 year:

Run command

$ openssl req -x509 -nodes -newkey rsa:4096 -keyout mykey.pem -out mycert.pem -days 365 -config <(
cat <<EOF
[req]
prompt = no
default_md = sha256
distinguished_name = dn

[ dn ]
C=US
ST=District of Columbia
L=Washington
O=General Services Administration
OU=Technology Transformation Service
[email protected]
CN=[domain-name-for-service-provider]

EOF
)

Replace the CN value [domain-name-for-service-provider] with the corresponding domain names of the apps, so that the certificates are recognizable. This command will generate two files: mykey.pem as the private key and mycert.pem as the public certificate.

Update SP public certificate on the login.gov dashboard

We have three Active apps and four Inactive apps on sandbox login.gov dashboard:

Remove the existing certificate and choose to upload file mycert.pem to have the app updated with the new public certificate. Changes made to Active apps take effect right away. The inactive ones need to be promoted to login.gov production via change requests. You can chat in Slack channel #login-partner-support for your request status update. Be aware that the promoting process is manual and error prone. Be very explicit about what values have been changed in which fields, since login.gov support is doing manual updating on the web UI. In most cases we only update the public certificate. Make sure to mention the old cert need to removed first, then add the new cert.

In the change request form, choose General Services Administration as Agency. Choose the following as Application names. The FCS in the names is there for legacy reason and they have to match what we have as the Signed Interagency Agreement (IAA) names.

  • Catalog@FCS (staging)
  • Data.gov Catalog
  • Inventory@FCS (staging)
  • Data.gov Inventory

Save to SP certiciates to team google drive.

Location: DatagovDevSecOps > secrets. The files are version controlled. Upload the certificate files as new version. This should be done before the next step so that other team members have access to the certificates to perform the next step.

Update SP certificates for cloud.gov apps:

In the code, replace the placeholder text the public certificates.

Run cf command cf env catalog-admin to get the credential value for service catalog-secrets. It is cf env inventory for inventory-secrets. For example

{
  "CKAN___BEAKER__SESSION__SECRET": "[...]",
  ...
  "SAML2_PRIVATE_KEY": "LS0[...]0tLQo="
}

Save it as file secrets.json in your local.

Run shell command base64 -w 0 mykey.pem > key-encoded.txt to encode the private key into an unwrapped string. Use the content of file key-encoded.txt to replace the value for SAML2_PRIVATE_KEY in file secrets.json.

Run cf command cf uups catalog-secrets -p secrets.json to update the cf service catalog-secrets. Restart the catalog-admin app to pick up the updated SAML2_PRIVATE_KEY value.

These steps need to be done 6 times to complete private key update for all apps Catalog and Inventory in cf spaces development, staging and prod.

This step has to be done together with login.gov dashboard certificate update. If one is done without the other, it means down time for the app login function. So when login.gov support notifies the task has been scheduled/resolved, it is time to do this step.

To update SP certificates for local docker Catalog app:

SP certificates go to ckan/saml2/pki/mycert.pem and ckan/saml2/pki/mykey.pem.

Other thoughts

  1. Once an app gets promoted to login.gov production server, we lost access to manage it, so any change including SP public certificate change needs to go through login.gov supporting channel. This might get improved in the future when login.gov production server supports developer self-management.
  2. login.gov works with self-assigned and expired SP certificates. That is why the SP certificate annual rotation is optional. It is NOT a login.gov SAML integration requirement.
  3. Any SP certificate change will cause some downtime, during which new login request won't be able to get authenticated. This is due to the difference between the time when our code gets deployed and when login.gov team updates the SP certificate. To avoid/minimize this downtime, we could have multiple login.gov apps running for each application, and alternate between them when new SP certificate is ready. But each app on login.gov incurs some expense. Or, we could reduce the frequency of SP update by giving new SP certificate longer expiry date, so SP certificate does not have to updated every year.
  4. According to login.gov support, if our SAML request is signed, we should be able have multiple certificates added to the dashboard and login.gov server will pick the correct one to respond. If this is the case, the correct SP rotation steps are: firstly have the login.gov with add new SP certificates in addition to the existing one, then change the CF environment variables in the CKAN app. Next year when make a new change request, remove the old one, keep the existing one as old one, then add the new one. This eliminates the downtime during SP certificate rotation. But our CKAN SAML request are not signed. So we need to get SAML request signed first then try this approach. For now, we need to be explicit in the request that we can only have one certificate. Login.gov support tends to add the new one in as multiple certificates, then the IdP always choose the 1st one (old one) as default for our non-signed SAML request.
  5. For local docker and development SP certificates, it is OK to give it a longer expiry date, say 10 years.
  6. Production CKAN applications should NOT use login.gov sandbox environment. Sandbox has no ATO, is not cleared for official use.
  7. The inactive apps on the login.gov sandbox dashboard are mirroring our login.gov production apps. It is important to keep their App name intact. They might not make sense (such as Catalog@FCS (staging)) but they are associated with the IAA numbers, that is how login.gov identify our apps in production.
Clone this wiki locally