Incident Report: Unauthorized BasicAuth activation on Beta environment

[placek]

Incident summary

• Title: Unauthorized BasicAuth activation on Beta environment
• Date/Time Reported: 2024-02-13T15:55 Local Time
• Reported By: Ryan (@Ryun1 [email protected])
• Incident Response Start: 2024-02-13T15:55 Local Time
• Resolution Achieved: 2024-02-13T16:18 Local Time
• Addressed By: Paweł (@placek [email protected])
• Affected Environment: beta
• Impact: Users were unable to access the application due to an unintended BasicAuth prompt.
• Related Tickets: Rebrand VVA to govtool in infrastructure configuration #97 (initiator), Enhance deployment script for conditional generation of BasicAuth configuration files #171 (resultant bug issue)

Incident description

During the deployment process aimed at the beta environment, an incident occurred where BasicAuth configuration from previous deployments (initiated for dev, test, and staging environments) was inadvertently reused. This misconfiguration led to BasicAuth being enabled on the beta environment unexpectedly. As a direct consequence, all user access to the application was blocked, including access to a maintenance page intended for use during operational work or issue resolution. Users attempting to access the application were met with BasicAuth credential prompts, effectively barring entry.

Detection and response

The issue was promptly detected and reported by @Ryun1 at 2024-02-13 15:55 local time. Immediate action was taken by @placek, who responded to the incident by removing the BasicAuth configuration on the beta environment. This action, completed by 2024-02-13 16:18 local time, successfully restored user access to the application. @Ryun1 confirmed the expected result of the action right after.

Note

Conversation on slack

Resolution and recovery

The hotfix implemented by @placek involved the direct removal of the BasicAuth configuration, which was not intended for the beta environment. This swift action reinstated normal access conditions for all users. Following the resolution, @placek reported bug issue #171 to document the incident and initiate a thorough review to prevent recurrence.

Root cause analysis

The root cause of the incident was identified as a flaw in the deployment script, which did not account for the existence of BasicAuth configuration files before generating new ones. This oversight led to the erroneous application of a previous environment's BasicAuth settings to the beta environment.

Actions and recommendations

• Immediate Action: Review and modify the deployment script to include checks for existing BasicAuth configuration files and correctly apply environment-specific conditions for their generation. See the issue Enhance deployment script for conditional generation of BasicAuth configuration files #171.
• Preventative Measures: Implement a validation step (aka smoke test) in the deployment process to ensure environment configurations, especially security features like BasicAuth, are correctly set according to predefined environment specifications.
• Documentation: Update deployment and incident response documentation to include this incident as a case study for training and improvement purposes.
• Follow-Up: Monitor the deployment process closely for the next few cycles to ensure the corrective measures are effective.

Conclusion

This incident underscores the importance of rigorous validation in deployment processes, especially when dealing with security features that can significantly impact user access. The swift response and resolution of this incident minimized potential disruptions. However, it also highlighted areas for improvement in deployment script robustness and process validation. Continued vigilance and improvement in these areas will be critical to preventing similar incidents in the future.