[Tiny EPIC] Support SOLR standalone on ECS

[jbrown-xentity]

Purpose

We want to a security compliant SOLR, but we're not sure how to do that.

Given above this need, conducting ECS deployment with SOLR8 image is needed to provide factual knowledge on future steps.

Acceptance Criteria

[ACs should be clearly demo-able/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN a CKAN system needs a SOLR instance
  WHEN 5 days have expired
  THEN CKAN can connect to SOLR
  AND an index is started
  AND the necessary stories are created to implement this in a production way

Background

To work around security issues around supporting EKS in AWS.
We already deployed SOLR8 on EKS in AWS, should be able to be ported to terraform.

Sketch

Feasibility Testing

• Convert James' DOI Solr deployment to terraform (in datagov-brokerpak-solr repo
  • Add Secure DNS
  • Add SSL
  • Add encryption (at-rest and in-transit) for EFS volume
  • Add authentication to Solr
  • Create a secure way of generating an initial Admin password
  • Enable logging for ECS services
  • Update appropriate IAM roles for ECS/EFS/LB/et cetera...
  • Add CloudMap Service Discovery for container-to-container dns communication
• On branch of catalog, move back to a single Solr deployment
• Push Solr Classic docker image to GHCR
• Iterate until bugs are fixed
• Connect to local/cloud.gov catalog instance (whichever is more convenient)
• Validate solr search index works or solr harvest works (whichever is more convenient)

Make service broker-able

• Package terraform code into brokerpak definition
• Rework processes and procedures to support AWS Credentials in repo
  • Copy local testing from eks-brokerpak
  • Setup Github Action secrets for CI tests to pass
• (Local) Make sure solr-on-ecs service can provision/bind and unbind/deprovision
• Preserve the original solrcloud service as a sibling service that the solr brokerpak supports
• (Local) Make sure solr-cloud service can still provision/bind and unbind/deprovision
• Get Github Action tests working
• Solr on ECS GSA-TTS/datagov-brokerpak-solr#36 (merge this)

Follow-on work

• Test reindex speed on a production-comparable DB size
• Test harvesting reliability
• Perform load testing to gauge the performance of solr standalone
• Make a new ticket for Leader-Follower paradigm (if necessary)
• Make a new ticket for EFS performance boosting (if necessary)

Example: Add Leader-Follower paradigm

• Make a new PR
• Add new ECS services for the followers
• Create script to initialize Follower containers
• Ensure Leader and Follower Solr instances can communicate

Example: Boost EFS performance

• Make a new PR
• Try setting an initial size for EFS volume
• Test Throughput options on EFS
• Test forcing a single AZ to see if there is a cross-availibility-zone issue
• Try Max IO Mode
• Try some suggestions here, https://docs.aws.amazon.com/efs/latest/ug/performance-tips.html

[nickumia-reisys]

List of references (in case we ever need them..):

• https://docs.aws.amazon.com/AmazonECS/latest/userguide/fargate-task-defs.html#fargate-tasks-size
• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bind-mounts.html
• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html#efs-add-content
• https://aws.amazon.com/premiumsupport/knowledge-center/ecs-fargate-mount-efs-containers-tasks/
• https://aws.amazon.com/premiumsupport/knowledge-center/fargate-unable-to-mount-efs/
• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html
• https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions-mountpoints.html
• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html
• https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/efs_access_point
• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/efs-volumes.html#specify-efs-config
• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-tutorials.html
• https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-register-targets.html
• https://github.com/docker-solr/docker-solr/blob/master/8.11/Dockerfile
• https://solr.apache.org/guide/6_6/solr-control-script-reference.html
• https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html
• https://stackoverflow.com/a/65968060
• https://www.terraform.io/language/expressions/for
• https://docs.aws.amazon.com/efs/latest/ug/iam-access-control-nfs-efs.html
• https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#target-group-routing-configuration
• https://aws.amazon.com/about-aws/whats-new/2020/05/network-load-balancer-now-supports-tls-alpn-policies/
• https://docs.aws.amazon.com/en_en/AmazonECS/latest/developerguide/using-service-linked-roles.html
• https://docs.aws.amazon.com/en_en/AmazonECS/latest/developerguide/troubleshoot-service-load-balancers.html
• https://medium.com/@ilia.lazebnik/attaching-an-efs-file-system-to-an-ecs-task-7bd15b76a6ef
• https://github.com/aws-containers/amazon-ecs-exec-checker
• Using aws_ecs_task_definition always wants to recreate the volume config for docker hashicorp/terraform-provider-aws#11796
• https://aws.amazon.com/blogs/containers/developers-guide-to-using-amazon-efs-with-amazon-ecs-and-aws-fargate-part-3/
• https://stackoverflow.com/questions/56403309/how-can-i-mount-an-efs-share-to-aws-fargate
• https://gist.github.com/henkin/7b58b3d52fe1e8b622fc7a147e0b88a5
• https://github.com/cloudfoundry/cloud-service-broker/blob/main/docs/brokerpak-specification.md
• https://github.com/cloudfoundry/csb-brokerpak-aws/
• https://www.terraform.io/language/functions/nonsensitive
• https://stackoverflow.com/a/1988265
• https://medium.com/@nataizya.s/understanding-srv-records-for-ecs-service-discovery-1d3b0ff64a4d
• https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/service_discovery_private_dns_namespace
• https://medium.com/containers-on-aws/how-to-setup-service-discovery-in-elastic-container-service-3d18479959e6
• https://serverfault.com/a/900860

[nickumia-reisys]

Two more that have to do with EFS encryption things:

• efs_volume_configuration transit_encryption_port causing replace of task definition  hashicorp/terraform-provider-aws#23541
• ECS Task Definition needs additional parameters for EFS Access Points hashicorp/terraform-provider-aws#13286

[nickumia-reisys]

As a retrospective comment to anyone who wanders onto this ticket (probably future-me 😅), this was a very important effort in the cloud migration of Data.gov apps, specifically https://catalog.data.gov and https://inventory.data.gov. I attempted to link all of the follow-on work that was needed after this pivotal ticket occurred that helped catapult this over the finish line; however, there's probably a few that still slipped through the cracks.

I consider this one of my biggest contributions to Data.gov. There were many possible paths for how Data.gov could procure a production Solr setup. Prior to me joining Data.gov, there was work to create Solr as an application on cloud.gov. This was made impractical because CloudFoundry only allows a maximum of 6GB of persistent storage and our Solr instance (as of writing) requires ~22GB. The next step was converting the app into a custom cloud.gov service based on Apache's solr-operator/solrcloud and AWS's EKS architecture. There were many forks and convoluted paths that we (@mogul, me and others before me) struggled with in its design. The EKS Brokerpak is still alive and very much practical for other projects. While this path never hit a hard wall or dead-end, the entire design was overly complex and had many, many moving parts. It was abandoned when there were a host of mystifying errors and bugs with indiscernible causes, specifically about solrcloud. It was at this point that inspiration from @jbrown-xentity led us to a pure Solr implementation on ECS (this ticket). This simplified our design on two fronts: (1) ECS had better defaults than EKS for us. It was like using docker-compose over kubernetes. (2) Data.gov was more familiar with solr than solrcloud. It had a Solr deployment on it's older platform and the team had better confidence Solr would be more stable.

In terms of how this was implemented, I don't really like it ( ...I know I wrote it ). As a relatively large user and producer of open-source code, Data.gov strives to stick closely with the communities we pull from and give back meaningful contributions as well. The less customization we have, the better we're able to develop, stay secure and remain integrated. Our Solr deployment is very special. It is at the intersection of many open-source communities ... Solr ... CKAN ... AWS ... Terraform ... Cloud.gov ... Brokerpaks ... (and maybe a few more) I feel like there were too many customizations to this code to meet the unspoken requirements of Data.gov. This is code we had to write because we couldn't borrow from something that existed already and it isn't well-abstracted for others to use it to do anything else. I believe this was a necessary evil for the position we were in, but going forward this will likely face a painful death in the future.

With more than 3 months of production catalog (and 4 months of production inventory) using this code, I think it's safe to say that it is rather successful. There were a few bugs and concerns after the initial release; but thanks to @FuhuXia's diligence, we've been able to monitor Solr's performance and health and ensure problems are taken care of. From the initial release of the Leader-Follower design, there has not been major changes to the core code or infrastructure. And I take that as a win.

I'm leaving this comment as a reminder for me and as counsel for whoever this code may effect in the future. This endeavor was an unwelcomingly large part of my life for almost a year. It wasn't very fun working on this. Did I learn a lot? Yes. Was it challenging? I don't think for the right reason haha, but yes. Probably, the only thing that got me through this was the encouragement, support and guidance that I received from my team. Very warm and hearty thanks to @mogul @jbrown-xentity @FuhuXia 🙇

Follow-on tickets:

• Add Leader-Follower paradigm to Solr-on-ECS #3854
• SOLR EFS Backups #3862
• Setup EFS Restore Procedures #3869
• SOLR throughput max out #3875
• Choose the right LockType for Solr leader/follower configuration #3920
• Restart SOLR when under memory pressure #3904