[SC-23] Ensure all inter-component traffic within Solr is sent over TLS

[nickumia-reisys]

User Story

In order to increase our security posture, the Data.gov Solr Team wants to secure all communication channels within our Solr deployment.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN communication between Solr and the LB is encrypted
  WHEN I look at the Solr service
  THEN I see it listening only on port 443

• GIVEN communication between Solr and the Admin Init app is encrypted
  WHEN I look at the logs of the Admin Init app
  THEN I see it talking to Solr over https

• GIVEN communication between Solr and EFS is encrypted
  WHEN I look at the Solr service
  THEN I see it talking to EFS in an encrypted way

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]
Related tickets:

• [Tiny EPIC] Support SOLR standalone on ECS #3826
• [datagov-brokerpak-solr] EFS in task definition does not encrypt data in transit  #4102

Security Considerations (required)

TODO

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

• Determine the scope of what traffic falls under this category
  • All internal traffic is unencrypted in the current design
    • Solr to/from LB
    • Solr to/from Admin Init app
    • Solr to/from EFS
• Design solutions for each component
• Implement them

[nickumia-reisys]

Full disclosure: This is not an easy issue. There are known issues as to why we couldn't get EFS encrypted-in-transit and to support HTTPS from the Solr node itself would require installing either self-signed certificates or some other solution which is further customization to the design (not to mention the implementation details themselves).

[hkdctol]

We don't know if this is possible to be implemented; may have to look into getting acceptance of current.

[btylerburton]

have things changed? is this easy now?
https://docs.aws.amazon.com/efs/latest/ug/encryption-in-transit.html

[nickumia-reisys]

So, I remembered that I looked at that page and that I tried to use the efs-utils. I was trying to track down the problem we saw when we tried to implement it. However, as a trail of that specific work @btylerburton:

• Commit that introduces it: GSA-TTS/datagov-brokerpak-solr@4cf4c5a
• Commit that removes it: GSA-TTS/datagov-brokerpak-solr@f18654d
• Commit that added the tools to the Solr image: GSA/catalog.data.gov@63faf92

There was some issue with connectivity. Either AWS didn't like the hack with efs-utils trying to initiate the connection after the Solr container was already running and since it couldn't establish the connection, Solr just made a temporary directory for its data. The primary issue was that the connection was not being established TLS and there were no errors, no warnings, no indications about which part of the connection was failing. Not being able to debug it, I had given up. We can try again, but I stand by the fact that it is not worth the effort to investigate this.

We had even submitted an issue with AWS Support and there wasn't any specific help that they could provide because they thought it was an application/infrastructure-specific issue that they were not knowledgeable enough to diagnose.

[hkdctol]

Next step looks like discuss with our ISSO.

[hkdctol]

Discussed with ISSO on 2/14/23. He will raise with ISSM.

[jbrown-xentity]

Summary doc @hkdctol

[hkdctol]

ISSO advises we start work on AOR, https://docs.google.com/document/d/18AvgMBln6fKgoznb3BWTvu2JHO7iISyf/edit?usp=sharing&ouid=113511966954817069922&rtpof=true&sd=true

[hkdctol]

Draft sent to ISSO

[hkdctol]

Met with ISSM 8/9, this is pending CISO's return from leave in a few weeks

[nickumia-reisys]

While the work wasn't fully documented in motion, an exhaustive search for how to enable TLS for EFS volumes in ECS was pursued combining documentation at the crossroads of Solr, Terraform and AWS. The full list of options were attempted without success setting up the IAM permissions, EFS volume, EFS access points, ECS mounting and solr volume setup. There may have been a bug or something very unique about our Solr application that was causing issues; however, all attempts were inconclusive.

All of the commits where in this PR:

• Solr on ECS GSA-TTS/datagov-brokerpak-solr#36
  • In the range of:
    • GSA-TTS/datagov-brokerpak-solr@d71744e
    • GSA-TTS/datagov-brokerpak-solr@a7f84a1

List of references used:

• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html#efs-add-content
• https://aws.amazon.com/premiumsupport/knowledge-center/ecs-fargate-mount-efs-containers-tasks/
• https://aws.amazon.com/premiumsupport/knowledge-center/fargate-unable-to-mount-efs/
• https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/efs_access_point
• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/efs-volumes.html#specify-efs-config
• https://docs.aws.amazon.com/efs/latest/ug/iam-access-control-nfs-efs.html
• https://medium.com/@ilia.lazebnik/attaching-an-efs-file-system-to-an-ecs-task-7bd15b76a6ef
• https://aws.amazon.com/blogs/containers/developers-guide-to-using-amazon-efs-with-amazon-ecs-and-aws-fargate-part-3/
• https://stackoverflow.com/questions/56403309/how-can-i-mount-an-efs-share-to-aws-fargate

AWS Support Ticket:

• https://support.console.aws.amazon.com/support/home?region=us-west-2#/case/?displayId=10114958361&language=en

[rpalmer-gsa]

@nickumia-reisys can you dump the support ticket to a PDF as I don't have access to view it.

Thank you

[nickumia-reisys]

@rpalmer-gsa Here is the doc version of the support ticket.

P.S. I tried my best to paste it in a readable structure. (additional discussion)

[hkdctol]

Met with ISSM 1/22, he will follow up on AOR status

[btylerburton]

Not carried over as a new POA&M