Root Cause Analysis
This incident was caused by human error where a database migration was initiated while a daily backup was still running on our production database. The overlap caused a conflict between the migration process and the backup operation.
Timeline
11:08 UTC - We are currently investigating reports of issues.
11:11 UTC - We received confirmations that users are no longer receiving server errors.
Resolution
To fix this issue, our team rescheduled the database migration to an earlier time, ensuring it does not coincide with the backup operation. Additionally, we updated our run book for database migrations to include a step to check if a backup is running before proceeding.
Remediation Items