I dropped our production database and now pay 10% more for AWS
alexeyondata.substack.com
March 6, 2026
16 min read
🔥🔥🔥🔥🔥
45/100
Summary
A Terraform command executed by an AI agent accidentally deleted the production infrastructure for the DataTalks.Club course platform, leading to a 10% increase in AWS costs during the restoration process. Plans include migrating the AI Shipping Labs website from static GitHub Pages to AWS and transitioning from a Next.js setup to a Django version.
Key Takeaways
- An AI agent executed a Terraform command that inadvertently wiped out the production infrastructure for the DataTalks.Club course platform, including all data stored for 2.5 years.
- The author upgraded to AWS Business Support for faster recovery assistance, which increased costs by 10%.
- The database was fully restored within 24 hours, containing over 1.9 million rows in the courses_answer table.
- Preventive measures implemented after the incident included setting up a backup Lambda function, enabling deletion protection, and creating S3 backups.
Community Sentiment
Positives
- The ability to recover deleted snapshots from AWS support is a rare and fortunate occurrence, highlighting the importance of backup strategies in production environments.
Concerns
- Relying on AI tools like Claude Code without proper oversight can lead to reckless decisions, as seen when the author approved a 'terraform destroy' command without understanding its implications.
- The integration of multiple projects into one workflow, despite warnings from the AI, demonstrates a lack of caution that can lead to significant operational risks.
- The incident underscores the critical need for a separate staging environment to prevent catastrophic mistakes in production settings, especially when dealing with valuable data.
