Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: API Audit Logs #824

Merged
merged 11 commits into from
Sep 16, 2024
Merged

feature: API Audit Logs #824

merged 11 commits into from
Sep 16, 2024

Conversation

bryan-robitaille
Copy link
Contributor

@bryan-robitaille bryan-robitaille commented Sep 13, 2024

Summary | Résumé

Enables API Audit Logging by:

  • Adding new DynamoDB table for API Audit Logs
  • Modifies the existing Audit Log Lambda to handle both App and API SQS audit log streams

Small Refactor:

  • Renames the outputs of DynamoDB and SQS modules to more clearly identify the different between the App and API Audit Log resources

Requires PR from forms-api to test properly: cds-snc/forms-api#54

closes cds-snc/platform-forms-client#4224

Test instructions | Instructions pour tester la modification

Locally run the App, API Server, and Localstack

  • Sign into the App, create a form, create an API Key for that form.
  • Generate some responses for the form through Preview or script
  • Retrieve responses using script in API server utils folder
  • Using NoSQL Workbench or another tool connect to the dynamodb and verify the audit log entries in both tables.

@bryan-robitaille bryan-robitaille marked this pull request as ready for review September 13, 2024 14:48
Copy link
Contributor

@craigzour craigzour left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going to review the API PR and do an end to end test before approving it.

About the archiving of API audit logs, is this something we want to implement later on?

aws/dynamodb/outputs.tf Outdated Show resolved Hide resolved
lambda-code/audit-logs/main.ts Show resolved Hide resolved
Copy link

⚠ Terrform update available

Terraform: 1.9.5 (using 1.9.2)
Terragrunt: 0.67.5 (using 0.63.2)

Copy link

Staging: sqs

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 2 to add, 0 to change, 0 to destroy
Show summary
CHANGE NAME
add aws_sqs_queue.api_audit_log_deadletter_queue
aws_sqs_queue.api_audit_log_queue
Show plan
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_sqs_queue.api_audit_log_deadletter_queue will be created
  + resource "aws_sqs_queue" "api_audit_log_deadletter_queue" {
      + arn                               = (known after apply)
      + content_based_deduplication       = false
      + deduplication_scope               = (known after apply)
      + delay_seconds                     = 60
      + fifo_queue                        = false
      + fifo_throughput_limit             = (known after apply)
      + id                                = (known after apply)
      + kms_data_key_reuse_period_seconds = 300
      + kms_master_key_id                 = "alias/aws/sqs"
      + max_message_size                  = 262144
      + message_retention_seconds         = 1209600
      + name                              = "api_audit_log_deadletter_queue"
      + name_prefix                       = (known after apply)
      + policy                            = (known after apply)
      + receive_wait_time_seconds         = 5
      + redrive_allow_policy              = (known after apply)
      + redrive_policy                    = (known after apply)
      + sqs_managed_sse_enabled           = (known after apply)
      + tags_all                          = {
          + "CostCentre" = "forms-platform-staging"
          + "Terraform"  = "true"
        }
      + url                               = (known after apply)
      + visibility_timeout_seconds        = 30
    }

  # aws_sqs_queue.api_audit_log_queue will be created
  + resource "aws_sqs_queue" "api_audit_log_queue" {
      + arn                               = (known after apply)
      + content_based_deduplication       = false
      + deduplication_scope               = (known after apply)
      + delay_seconds                     = 0
      + fifo_queue                        = false
      + fifo_throughput_limit             = (known after apply)
      + id                                = (known after apply)
      + kms_data_key_reuse_period_seconds = 300
      + kms_master_key_id                 = "alias/aws/sqs"
      + max_message_size                  = 262144
      + message_retention_seconds         = 172800
      + name                              = "api_audit_log_queue"
      + name_prefix                       = (known after apply)
      + policy                            = (known after apply)
      + receive_wait_time_seconds         = 0
      + redrive_allow_policy              = (known after apply)
      + redrive_policy                    = (known after apply)
      + sqs_managed_sse_enabled           = (known after apply)
      + tags_all                          = {
          + "CostCentre" = "forms-platform-staging"
          + "Terraform"  = "true"
        }
      + url                               = (known after apply)
      + visibility_timeout_seconds        = 1960
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + sqs_api_audit_log_deadletter_queue_arn = (known after apply)
  + sqs_api_audit_log_queue_arn            = (known after apply)
  + sqs_api_audit_log_queue_id             = (known after apply)
  + sqs_app_audit_log_deadletter_queue_arn = "arn:aws:sqs:ca-central-1:687401027353:audit_log_deadletter_queue"
  + sqs_app_audit_log_queue_arn            = "arn:aws:sqs:ca-central-1:687401027353:audit_log_queue"
  + sqs_app_audit_log_queue_id             = "https://sqs.ca-central-1.amazonaws.com/687401027353/audit_log_queue"
  - sqs_audit_log_deadletter_queue_arn     = "arn:aws:sqs:ca-central-1:687401027353:audit_log_deadletter_queue" -> null
  - sqs_audit_log_queue_arn                = "arn:aws:sqs:ca-central-1:687401027353:audit_log_queue" -> null
  - sqs_audit_log_queue_id                 = "https://sqs.ca-central-1.amazonaws.com/687401027353/audit_log_queue" -> null

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"
Show Conftest results
WARN - plan.json - main - Missing Common Tags: ["aws_sqs_queue.api_audit_log_deadletter_queue"]
WARN - plan.json - main - Missing Common Tags: ["aws_sqs_queue.api_audit_log_queue"]
WARN - plan.json - main - Missing Common Tags: ["aws_sqs_queue.audit_log_deadletter_queue"]
WARN - plan.json - main - Missing Common Tags: ["aws_sqs_queue.audit_log_queue"]
WARN - plan.json - main - Missing Common Tags: ["aws_sqs_queue.reliability_deadletter_queue"]
WARN - plan.json - main - Missing Common Tags: ["aws_sqs_queue.reliability_queue"]
WARN - plan.json - main - Missing Common Tags: ["aws_sqs_queue.reprocess_submission_queue"]

26 tests, 19 passed, 7 warnings, 0 failures, 0 exceptions

Copy link

Staging: dynamodb

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 1 to add, 0 to change, 0 to destroy
Show summary
CHANGE NAME
add aws_dynamodb_table.api_audit_logs
Show plan
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_dynamodb_table.api_audit_logs will be created
  + resource "aws_dynamodb_table" "api_audit_logs" {
      + arn                         = (known after apply)
      + billing_mode                = "PAY_PER_REQUEST"
      + deletion_protection_enabled = true
      + hash_key                    = "UserID"
      + id                          = (known after apply)
      + name                        = "ApiAuditLogs"
      + range_key                   = "Event#SubjectID#TimeStamp"
      + read_capacity               = (known after apply)
      + stream_arn                  = (known after apply)
      + stream_enabled              = false
      + stream_label                = (known after apply)
      + stream_view_type            = (known after apply)
      + tags_all                    = {
          + "CostCentre" = "forms-platform-staging"
          + "Terraform"  = "true"
        }
      + write_capacity              = (known after apply)

      + attribute {
          + name = "Event#SubjectID#TimeStamp"
          + type = "S"
        }
      + attribute {
          + name = "Status"
          + type = "S"
        }
      + attribute {
          + name = "TimeStamp"
          + type = "N"
        }
      + attribute {
          + name = "UserID"
          + type = "S"
        }

      + global_secondary_index {
          + hash_key           = "Status"
          + name               = "StatusByTimestamp"
          + non_key_attributes = []
          + projection_type    = "ALL"
          + range_key          = "TimeStamp"
        }
      + global_secondary_index {
          + hash_key           = "UserID"
          + name               = "UserByTime"
          + non_key_attributes = []
          + projection_type    = "KEYS_ONLY"
          + range_key          = "TimeStamp"
        }

      + point_in_time_recovery {
          + enabled = true
        }

      + server_side_encryption {
          + enabled     = true
          + kms_key_arn = "arn:aws:kms:ca-central-1:687401027353:key/1f3edb85-9eac-4da9-8c7c-43a68e339ede"
        }

      + ttl (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + dynamodb_api_audit_logs_arn        = (known after apply)
  + dynamodb_api_audit_logs_table_name = "ApiAuditLogs"
  + dynamodb_app_audit_logs_arn        = "arn:aws:dynamodb:ca-central-1:687401027353:table/AuditLogs"
  + dynamodb_app_audit_logs_table_name = "AuditLogs"
  - dynamodb_audit_logs_arn            = "arn:aws:dynamodb:ca-central-1:687401027353:table/AuditLogs" -> null
  - dynamodb_audit_logs_table_name     = "AuditLogs" -> null

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"
Show Conftest results
WARN - plan.json - main - Missing Common Tags: ["aws_dynamodb_table.api_audit_logs"]
WARN - plan.json - main - Missing Common Tags: ["aws_dynamodb_table.audit_logs"]
WARN - plan.json - main - Missing Common Tags: ["aws_dynamodb_table.reliability_queue"]
WARN - plan.json - main - Missing Common Tags: ["aws_dynamodb_table.vault"]

23 tests, 19 passed, 4 warnings, 0 failures, 0 exceptions

@bryan-robitaille
Copy link
Contributor Author

@craigzour

About the archiving of API audit logs, is this something we want to implement later on?

Yes, we can tackle that in a separate PR.

Copy link

Staging: app

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 1 to add, 1 to change, 0 to destroy
Show summary
CHANGE NAME
update aws_iam_policy.forms_sqs
add aws_ecs_task_definition.form_viewer
Show plan
Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # aws_ecs_task_definition.form_viewer will be created
  + resource "aws_ecs_task_definition" "form_viewer" {
      + arn                      = (known after apply)
      + arn_without_revision     = (known after apply)
      + container_definitions    = jsonencode(
            [
              + {
                  + Command                = null
                  + Cpu                    = 0
                  + CredentialSpecs        = null
                  + DependsOn              = null
                  + DisableNetworking      = null
                  + DnsSearchDomains       = null
                  + DnsServers             = null
                  + DockerLabels           = null
                  + DockerSecurityOptions  = null
                  + EntryPoint             = null
                  + Environment            = [
                      + {
                          + Name  = "AUDIT_LOG_QUEUE_URL"
                          + Value = "http://sqs.ca-central-1.localhost.localstack.cloud:4566/000000000000/audit_log_queue"
                        },
                      + {
                          + Name  = "COGNITO_CLIENT_ID"
                          + Value = "17bsg3b2b7q5snon007rru264u"
                        },
                      + {
                          + Name  = "COGNITO_ENDPOINT_URL"
                          + Value = "cognito-idp.ca-central-1.amazonaws.com/ca-central-1_Cguq9JNQ1"
                        },
                      + {
                          + Name  = "EMAIL_ADDRESS_CONTACT_US"
                          + Value = "[email protected]"
                        },
                      + {
                          + Name  = "EMAIL_ADDRESS_SUPPORT"
                          + Value = "[email protected]"
                        },
                      + {
                          + Name  = "HOST_URL"
                          + Value = "https://forms-staging.cdssandbox.xyz"
                        },
                      + {
                          + Name  = "METRIC_PROVIDER"
                          + Value = "stdout"
                        },
                      + {
                          + Name  = "NEXTAUTH_URL"
                          + Value = "https://forms-staging.cdssandbox.xyz"
                        },
                      + {
                          + Name  = "RECAPTCHA_V3_SITE_KEY"
                          + Value = "6LfJDN4eAAAAAGvdRF7ZnQ7ciqdo1RQnQDFmh0VY"
                        },
                      + {
                          + Name  = "REDIS_URL"
                          + Value = "gcforms-redis-rep-group.uwpetx.ng.0001.cac1.cache.amazonaws.com"
                        },
                      + {
                          + Name  = "RELIABILITY_FILE_STORAGE"
                          + Value = "forms-staging-reliability-file-storage"
                        },
                      + {
                          + Name  = "REPROCESS_SUBMISSION_QUEUE_URL"
                          + Value = "https://sqs.ca-central-1.amazonaws.com/687401027353/reprocess_submission_queue.fifo"
                        },
                      + {
                          + Name  = "TEMPLATE_ID"
                          + Value = "8d597a1b-a1d6-4e3c-8421-042a2b4158b7"
                        },
                      + {
                          + Name  = "TEMPORARY_TOKEN_TEMPLATE_ID"
                          + Value = "b6885d06-d10a-422a-973f-05e274d9aa86"
                        },
                      + {
                          + Name  = "TRACER_PROVIDER"
                          + Value = "stdout"
                        },
                      + {
                          + Name  = "VAULT_FILE_STORAGE"
                          + Value = "forms-staging-vault-file-storage"
                        },
                      + {
                          + Name  = "ZITADEL_PROVIDER"
                          + Value = "https://auth.forms-staging.cdssandbox.xyz"
                        },
                    ]
                  + EnvironmentFiles       = null
                  + Essential              = true
                  + ExtraHosts             = null
                  + FirelensConfiguration  = null
                  + HealthCheck            = null
                  + Hostname               = null
                  + Image                  = "687401027353.dkr.ecr.ca-central-1.amazonaws.com/form_viewer_staging"
                  + Interactive            = null
                  + Links                  = null
                  + LinuxParameters        = {
                      + Capabilities       = {
                          + Add  = []
                          + Drop = [
                              + "ALL",
                            ]
                        }
                      + Devices            = null
                      + InitProcessEnabled = null
                      + MaxSwap            = null
                      + SharedMemorySize   = null
                      + Swappiness         = null
                      + Tmpfs              = null
                    }
                  + LogConfiguration       = {
                      + LogDriver     = "awslogs"
                      + Options       = {
                          + awslogs-group         = "Forms"
                          + awslogs-region        = "ca-central-1"
                          + awslogs-stream-prefix = "ecs-form-viewer"
                        }
                      + SecretOptions = null
                    }
                  + Memory                 = null
                  + MemoryReservation      = null
                  + MountPoints            = []
                  + Name                   = "form_viewer"
                  + PortMappings           = [
                      + {
                          + AppProtocol        = ""
                          + ContainerPort      = 3000
                          + ContainerPortRange = null
                          + HostPort           = 3000
                          + Name               = null
                          + Protocol           = "tcp"
                        },
                    ]
                  + Privileged             = null
                  + PseudoTerminal         = null
                  + ReadonlyRootFilesystem = null
                  + RepositoryCredentials  = null
                  + ResourceRequirements   = null
                  + Secrets                = [
                      + {
                          + Name      = "DATABASE_URL"
                          + ValueFrom = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:server-database-url-0PSpE3"
                        },
                      + {
                          + Name      = "FRESHDESK_API_KEY"
                          + ValueFrom = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:freshdesk_api_key-JVyxop"
                        },
                      + {
                          + Name      = "GC_NOTIFY_CALLBACK_BEARER_TOKEN"
                          + ValueFrom = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:notify_callback_bearer_token-aXJPLs"
                        },
                      + {
                          + Name      = "NOTIFY_API_KEY"
                          + ValueFrom = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:notify_api_key-eR3nNp"
                        },
                      + {
                          + Name      = "RECAPTCHA_V3_SECRET_KEY"
                          + ValueFrom = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:recaptcha_secret-tTjsBo"
                        },
                      + {
                          + Name      = "SENTRY_API_KEY"
                          + ValueFrom = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:sentry_api_key-QBmONz"
                        },
                      + {
                          + Name      = "TOKEN_SECRET"
                          + ValueFrom = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:token_secret-n5Doyu"
                        },
                      + {
                          + Name      = "ZITADEL_ADMINISTRATION_KEY"
                          + ValueFrom = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:zitadel_administration_key-Oaki1d"
                        },
                    ]
                  + StartTimeout           = null
                  + StopTimeout            = null
                  + SystemControls         = []
                  + Ulimits                = null
                  + User                   = null
                  + VolumesFrom            = []
                  + WorkingDirectory       = null
                },
            ]
        )
      + cpu                      = "2048"
      + execution_role_arn       = "arn:aws:iam::687401027353:role/form-viewer"
      + family                   = "form-viewer"
      + id                       = (known after apply)
      + memory                   = "4096"
      + network_mode             = "awsvpc"
      + requires_compatibilities = [
          + "FARGATE",
        ]
      + revision                 = (known after apply)
      + skip_destroy             = false
      + tags_all                 = {
          + "CostCentre" = "forms-platform-staging"
          + "Terraform"  = "true"
        }
      + task_role_arn            = "arn:aws:iam::687401027353:role/form-viewer"
      + track_latest             = false
    }

  # aws_iam_policy.forms_sqs will be updated in-place
  ~ resource "aws_iam_policy" "forms_sqs" {
        id               = "arn:aws:iam::687401027353:policy/forms_sqs"
        name             = "forms_sqs"
      ~ policy           = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Resource = [
                            "arn:aws:sqs:ca-central-1:687401027353:reprocess_submission_queue.fifo",
                          ~ "arn:aws:sqs:ca-central-1:687401027353:audit_log_queue" -> "arn:aws:sqs:ca-central-1:000000000000:audit_log_queue",
                        ]
                      - Sid      = ""
                        # (2 unchanged attributes hidden)
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        tags             = {}
        # (7 unchanged attributes hidden)
    }

Plan: 1 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"
Show Conftest results
WARN - plan.json - main - Missing Common Tags: ["aws_appautoscaling_target.forms[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.forms"]
WARN - plan.json - main - Missing Common Tags: ["aws_codedeploy_app.app"]
WARN - plan.json - main - Missing Common Tags: ["aws_codedeploy_deployment_group.app"]
WARN - plan.json - main - Missing Common Tags: ["aws_ecs_cluster.forms"]
WARN - plan.json - main - Missing Common Tags: ["aws_ecs_service.form_viewer"]
WARN - plan.json - main - Missing Common Tags: ["aws_ecs_task_definition.form_viewer"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.cognito"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.forms_dynamodb"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.forms_kms"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.forms_s3"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.forms_secrets_manager"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.forms_sqs"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.codedeploy"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.forms"]

34 tests, 19 passed, 15 warnings, 0 failures, 0 exceptions

Copy link

Staging: api

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 0 to add, 2 to change, 0 to destroy
Show summary
CHANGE NAME
update module.api_ecs.aws_ecs_service.this
module.api_ecs.aws_iam_policy.this_task[0]
Show plan
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.api_ecs.aws_ecs_service.this will be updated in-place
  ~ resource "aws_ecs_service" "this" {
        id                                 = "arn:aws:ecs:ca-central-1:687401027353:service/Forms/forms-api"
        name                               = "forms-api"
        tags                               = {
            "CostCentre" = "forms-platform-staging"
            "Terraform"  = "true"
        }
      ~ task_definition                    = "forms-api:43" -> "forms-api"
        # (15 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

  # module.api_ecs.aws_iam_policy.this_task[0] will be updated in-place
  ~ resource "aws_iam_policy" "this_task" {
        id               = "arn:aws:iam::687401027353:policy/forms-api_ecs_task_policy"
        name             = "forms-api_ecs_task_policy"
      ~ policy           = jsonencode(
          ~ {
              ~ Statement = [
                    # (3 unchanged elements hidden)
                    {
                        Action   = "secretsmanager:GetSecretValue"
                        Effect   = "Allow"
                        Resource = "arn:aws:secretsmanager:ca-central-1:687401027353:secret:server-database-url-0PSpE3"
                    },
                  + {
                      + Action   = [
                          + "sqs:SendMessage",
                          + "sqs:GetQueueUrl",
                        ]
                      + Effect   = "Allow"
                      + Resource = "arn:aws:sqs:ca-central-1:000000000000:api_audit_log_queue"
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        tags             = {
            "CostCentre" = "forms-platform-staging"
            "Terraform"  = "true"
        }
        # (7 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"
Show Conftest results
20 tests, 20 passed, 0 warnings, 0 failures, 0 exceptions

Copy link

Staging: lambdas

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

⚠️   Warning: resources will be destroyed by this change!

Plan: 2 to add, 2 to change, 1 to destroy
Show summary
CHANGE NAME
add aws_lambda_event_source_mapping.api_audit_logs
aws_lambda_event_source_mapping.app_audit_logs
delete aws_lambda_event_source_mapping.audit_logs
update aws_iam_policy.lambda_dynamodb
aws_lambda_function.audit_logs
Show plan
Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
  - destroy

Terraform will perform the following actions:

  # aws_iam_policy.lambda_dynamodb will be updated in-place
  ~ resource "aws_iam_policy" "lambda_dynamodb" {
        id               = "arn:aws:iam::687401027353:policy/lambda_dynamobdb"
        name             = "lambda_dynamobdb"
      ~ policy           = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Resource = [
                            # (3 unchanged elements hidden)
                            "arn:aws:dynamodb:ca-central-1:687401027353:table/ReliabilityQueue",
                          - "arn:aws:dynamodb:ca-central-1:687401027353:table/AuditLogs/index/*",
                          - "arn:aws:dynamodb:ca-central-1:687401027353:table/AuditLogs",
                          + "arn:aws:dynamodb:ca-central-1:123456789012:table/AuditLogs/index/*",
                          + "arn:aws:dynamodb:ca-central-1:123456789012:table/AuditLogs",
                          + "arn:aws:dynamodb:ca-central-1:123456789012:table/ApiAuditLogs/index/*",
                          + "arn:aws:dynamodb:ca-central-1:123456789012:table/ApiAuditLogs",
                        ]
                      - Sid      = ""
                        # (2 unchanged attributes hidden)
                    },
                ]
                # (1 unchanged attribute hidden)
            }
        )
        tags             = {}
        # (7 unchanged attributes hidden)
    }

  # aws_lambda_event_source_mapping.api_audit_logs will be created
  + resource "aws_lambda_event_source_mapping" "api_audit_logs" {
      + batch_size                         = 10
      + enabled                            = true
      + event_source_arn                   = "arn:aws:sqs:ca-central-1:000000000000:api_audit_log_queue"
      + function_arn                       = (known after apply)
      + function_name                      = "arn:aws:lambda:ca-central-1:687401027353:function:audit-logs"
      + function_response_types            = [
          + "ReportBatchItemFailures",
        ]
      + id                                 = (known after apply)
      + last_modified                      = (known after apply)
      + last_processing_result             = (known after apply)
      + maximum_batching_window_in_seconds = 30
      + maximum_record_age_in_seconds      = (known after apply)
      + maximum_retry_attempts             = (known after apply)
      + parallelization_factor             = (known after apply)
      + state                              = (known after apply)
      + state_transition_reason            = (known after apply)
      + uuid                               = (known after apply)

      + amazon_managed_kafka_event_source_config (known after apply)

      + self_managed_kafka_event_source_config (known after apply)
    }

  # aws_lambda_event_source_mapping.app_audit_logs will be created
  + resource "aws_lambda_event_source_mapping" "app_audit_logs" {
      + batch_size                         = 10
      + enabled                            = true
      + event_source_arn                   = "arn:aws:sqs:ca-central-1:000000000000:audit_log_queue"
      + function_arn                       = (known after apply)
      + function_name                      = "arn:aws:lambda:ca-central-1:687401027353:function:audit-logs"
      + function_response_types            = [
          + "ReportBatchItemFailures",
        ]
      + id                                 = (known after apply)
      + last_modified                      = (known after apply)
      + last_processing_result             = (known after apply)
      + maximum_batching_window_in_seconds = 30
      + maximum_record_age_in_seconds      = (known after apply)
      + maximum_retry_attempts             = (known after apply)
      + parallelization_factor             = (known after apply)
      + state                              = (known after apply)
      + state_transition_reason            = (known after apply)
      + uuid                               = (known after apply)

      + amazon_managed_kafka_event_source_config (known after apply)

      + self_managed_kafka_event_source_config (known after apply)
    }

  # aws_lambda_event_source_mapping.audit_logs will be destroyed
  # (because aws_lambda_event_source_mapping.audit_logs is not in configuration)
  - resource "aws_lambda_event_source_mapping" "audit_logs" {
      - batch_size                         = 10 -> null
      - bisect_batch_on_function_error     = false -> null
      - enabled                            = true -> null
      - event_source_arn                   = "arn:aws:sqs:ca-central-1:687401027353:audit_log_queue" -> null
      - function_arn                       = "arn:aws:lambda:ca-central-1:687401027353:function:audit-logs" -> null
      - function_name                      = "arn:aws:lambda:ca-central-1:687401027353:function:audit-logs" -> null
      - function_response_types            = [
          - "ReportBatchItemFailures",
        ] -> null
      - id                                 = "bc90f6e4-30dc-4493-a000-2635f5db0fe9" -> null
      - last_modified                      = "2024-05-02T15:22:41Z" -> null
      - maximum_batching_window_in_seconds = 30 -> null
      - maximum_record_age_in_seconds      = 0 -> null
      - maximum_retry_attempts             = 0 -> null
      - parallelization_factor             = 0 -> null
      - queues                             = [] -> null
      - state                              = "Enabled" -> null
      - state_transition_reason            = "USER_INITIATED" -> null
      - topics                             = [] -> null
      - tumbling_window_in_seconds         = 0 -> null
      - uuid                               = "bc90f6e4-30dc-4493-a000-2635f5db0fe9" -> null
        # (3 unchanged attributes hidden)
    }

  # aws_lambda_function.audit_logs will be updated in-place
  ~ resource "aws_lambda_function" "audit_logs" {
        id                             = "audit-logs"
        tags                           = {}
        # (28 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              + "API_AUDIT_LOGS_SQS_ARN" = "arn:aws:sqs:ca-central-1:000000000000:api_audit_log_queue"
              + "APP_AUDIT_LOGS_SQS_ARN" = "arn:aws:sqs:ca-central-1:000000000000:audit_log_queue"
                # (2 unchanged elements hidden)
            }
        }

        # (3 unchanged blocks hidden)
    }

Plan: 2 to add, 2 to change, 1 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"
Show Conftest results
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.audit_logs_archiver_lambda_trigger"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.form_archiver_lambda_trigger"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.nagware_lambda_trigger"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.reliability_dlq_lambda_trigger"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.response_archiver_lambda_trigger"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.archive_form_templates"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.audit_logs"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.audit_logs_archiver"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.dead_letter_queue_consumer"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.nagware"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.reliability"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.response_archiver"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.submission"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.vault_integrity"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_dynamodb"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_kms"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_logging"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_rds"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_s3"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_secrets"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_sns"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.lambda_sqs"]
WARN -...

Copy link

Staging: alarms

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 1 to add, 2 to change, 0 to destroy
Show summary
CHANGE NAME
add aws_cloudwatch_metric_alarm.api_audit_log_dead_letter_queue_warn
update aws_cloudwatch_metric_alarm.audit_log_dead_letter_queue_warn
aws_iam_role_policy.athena_dynamodb_policy
Show plan
Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # aws_cloudwatch_metric_alarm.api_audit_log_dead_letter_queue_warn will be created
  + resource "aws_cloudwatch_metric_alarm" "api_audit_log_dead_letter_queue_warn" {
      + actions_enabled                       = true
      + alarm_actions                         = [
          + "arn:aws:sns:ca-central-1:687401027353:alert-warning",
        ]
      + alarm_description                     = "Detect when a message is sent to the API Audit Log Dead Letter Queue"
      + alarm_name                            = "ApiAuditLogDeadLetterQueueWarn"
      + arn                                   = (known after apply)
      + comparison_operator                   = "GreaterThanThreshold"
      + evaluate_low_sample_count_percentiles = (known after apply)
      + evaluation_periods                    = 1
      + id                                    = (known after apply)
      + tags_all                              = {
          + "CostCentre" = "forms-platform-staging"
          + "Terraform"  = "true"
        }
      + threshold                             = 0
      + treat_missing_data                    = "missing"

      + metric_query {
          + id          = "m1"
          + return_data = false
            # (3 unchanged attributes hidden)

          + metric {
              + dimensions  = {
                  + "QueueName" = "arn:aws:sqs:ca-central-1:000000000000:api_audit_log_deadletter_queue"
                }
              + metric_name = "ApproximateNumberOfMessagesVisible"
              + namespace   = "AWS/SQS"
              + period      = 60
              + stat        = "Sum"
              + unit        = "Count"
            }
        }
      + metric_query {
          + id          = "m2"
          + return_data = false
            # (3 unchanged attributes hidden)

          + metric {
              + dimensions  = {
                  + "QueueName" = "arn:aws:sqs:ca-central-1:000000000000:api_audit_log_deadletter_queue"
                }
              + metric_name = "ApproximateNumberOfMessagesNotVisible"
              + namespace   = "AWS/SQS"
              + period      = 60
              + stat        = "Sum"
              + unit        = "Count"
            }
        }
      + metric_query {
          + expression  = "RATE(m2+m1)"
          + id          = "e1"
          + label       = "Error Rate"
          + return_data = true
            # (1 unchanged attribute hidden)
        }
    }

  # aws_cloudwatch_metric_alarm.audit_log_dead_letter_queue_warn will be updated in-place
  ~ resource "aws_cloudwatch_metric_alarm" "audit_log_dead_letter_queue_warn" {
        id                                    = "AuditLogDeadLetterQueueWarn"
        tags                                  = {}
        # (22 unchanged attributes hidden)

      - metric_query {
          - id          = "m1" -> null
          - period      = 0 -> null
          - return_data = false -> null
            # (3 unchanged attributes hidden)

          - metric {
              - dimensions  = {
                  - "QueueName" = "arn:aws:sqs:ca-central-1:687401027353:audit_log_deadletter_queue"
                } -> null
              - metric_name = "ApproximateNumberOfMessagesVisible" -> null
              - namespace   = "AWS/SQS" -> null
              - period      = 60 -> null
              - stat        = "Sum" -> null
              - unit        = "Count" -> null
            }
        }
      - metric_query {
          - id          = "m2" -> null
          - period      = 0 -> null
          - return_data = false -> null
            # (3 unchanged attributes hidden)

          - metric {
              - dimensions  = {
                  - "QueueName" = "arn:aws:sqs:ca-central-1:687401027353:audit_log_deadletter_queue"
                } -> null
              - metric_name = "ApproximateNumberOfMessagesNotVisible" -> null
              - namespace   = "AWS/SQS" -> null
              - period      = 60 -> null
              - stat        = "Sum" -> null
              - unit        = "Count" -> null
            }
        }
      - metric_query {
          - expression  = "RATE(m2+m1)" -> null
          - id          = "e1" -> null
          - label       = "Error Rate" -> null
          - period      = 0 -> null
          - return_data = true -> null
            # (1 unchanged attribute hidden)
        }
      + metric_query {
          + id          = "m1"
          + return_data = false
            # (3 unchanged attributes hidden)

          + metric {
              + dimensions  = {
                  + "QueueName" = "arn:aws:sqs:ca-central-1:000000000000:audit_log_deadletter_queue"
                }
              + metric_name = "ApproximateNumberOfMessagesVisible"
              + namespace   = "AWS/SQS"
              + period      = 60
              + stat        = "Sum"
              + unit        = "Count"
            }
        }
      + metric_query {
          + id          = "m2"
          + return_data = false
            # (3 unchanged attributes hidden)

          + metric {
              + dimensions  = {
                  + "QueueName" = "arn:aws:sqs:ca-central-1:000000000000:audit_log_deadletter_queue"
                }
              + metric_name = "ApproximateNumberOfMessagesNotVisible"
              + namespace   = "AWS/SQS"
              + period      = 60
              + stat        = "Sum"
              + unit        = "Count"
            }
        }
      + metric_query {
          + expression  = "RATE(m2+m1)"
          + id          = "e1"
          + label       = "Error Rate"
          + return_data = true
        }
    }

  # aws_iam_role_policy.athena_dynamodb_policy will be updated in-place
  ~ resource "aws_iam_role_policy" "athena_dynamodb_policy" {
        id          = "athena_dynamodb_role:athena_dynamodb_policy"
        name        = "athena_dynamodb_policy"
      ~ policy      = jsonencode(
          ~ {
              ~ Statement = [
                    # (1 unchanged element hidden)
                    {
                        Action   = [
                            "glue:GetTableVersions",
                            "glue:GetPartitions",
                            "glue:GetTables",
                            "glue:GetTableVersion",
                            "glue:GetDatabases",
                            "glue:GetTable",
                            "glue:GetPartition",
                            "glue:GetDatabase",
                            "glue:ListSchemas",
                            "athena:GetQueryExecution",
                            "s3:ListAllMyBuckets",
                        ]
                        Effect   = "Allow"
                        Resource = "*"
                    },
                  ~ {
                      ~ Resource = [
                          ~ "arn:aws:dynamodb:ca-central-1:687401027353:table/AuditLogs" -> "arn:aws:dynamodb:ca-central-1:123456789012:table/AuditLogs",
                          ~ "arn:aws:dynamodb:ca-central-1:687401027353:table/auditlogs" -> "arn:aws:dynamodb:ca-central-1:123456789012:table/auditlogs",
                        ]
                        # (2 unchanged attributes hidden)
                    },
                    {
                        Action   = [
                            "dynamodb:ListTables",
                        ]
                        Effect   = "Allow"
                        Resource = "*"
                    },
                    # (2 unchanged elements hidden)
                ]
                # (1 unchanged attribute hidden)
            }
        )
        # (2 unchanged attributes hidden)
    }

Plan: 1 to add, 2 to change, 0 to destroy.

Warning: Argument is deprecated

  with module.athena_bucket.aws_s3_bucket.this,
  on .terraform/modules/athena_bucket/S3/main.tf line 8, in resource "aws_s3_bucket" "this":
   8: resource "aws_s3_bucket" "this" {

Use the aws_s3_bucket_server_side_encryption_configuration resource instead

(and 3 more similar warnings elsewhere)

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"
Show Conftest results
WARN - plan.json - main - Missing Common Tags: ["aws_athena_data_catalog.dynamodb"]
WARN - plan.json - main - Missing Common Tags: ["aws_athena_data_catalog.rds_data_catalog"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.codedeploy_sns"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.notify_slack"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.ELB_5xx_error_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.ELB_healthy_hosts"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.UnHealthyHostCount-TargetGroup1"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.UnHealthyHostCount-TargetGroup2"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.alb_ddos"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.api_audit_log_dead_letter_queue_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.api_cpu_utilization_high_warn[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.api_lb_healthy_host_count[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.api_lb_unhealthy_host_count[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.api_memory_utilization_high_warn[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.api_response_time_warn[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.audit_log_dead_letter_queue_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.cognito_login_outside_canada_warn"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.cognito_signin_exceeded"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.ddos_detected_forms_warn"]
WARN - plan.json - main - Missing Common...

@bryan-robitaille bryan-robitaille merged commit 401d6b9 into develop Sep 16, 2024
11 checks passed
@bryan-robitaille bryan-robitaille deleted the feature/api_audit_logs branch September 16, 2024 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement audit logs
3 participants