Data loss is the worst that can happen to a company. It immediately destroys customers’ trust and leads to many technical issues. There is a way to avoid that with database guardrails. Let’s see how and why database guardrails could have saved Atlassian from their data loss.
Tests Do Not Capture Your Issues
Databases are prone to various failures but often don’t undergo the same rigorous testing as applications. Developers usually confirm that applications can read and write data accurately, but the methods behind these operations are frequently overlooked. Essential aspects like proper indexing, avoiding excessive lazy loading, and optimizing query efficiency are often neglected. For example, while we may validate the number of rows returned by a query, we might overlook the number of rows processed to get that result. Moreover, rollback procedures are rarely tested, exposing systems to potential data loss with every modification. To mitigate these risks, implementing robust automated tests is crucial. These tests can proactively identify issues, minimizing the need for manual intervention.
Load tests are commonly used to identify performance issues, but they come with notable drawbacks. While they can verify if queries are ready for production, they are costly to build and maintain. Ensuring GDPR compliance, data anonymization, and proper state management adds to the complexity. More importantly, load tests are conducted late in the development process, often after changes have been implemented, reviewed, and merged. This timing forces teams to backtrack and redo work, significantly slowing progress. Additionally, load tests are time-consuming, often requiring hours to warm up caches and confirm application reliability, making them unsuitable for catching issues early in development.
Another frequent challenge is testing with databases that are too small to expose performance issues early on. This shortfall not only delays the detection of bottlenecks during load testing but also leaves critical areas like schema migrations untested. Consequently, development slows, critical issues emerge, and overall agility suffers.
Yet, there is another overlooked issue.
Data Loss and How It Affected Atlassian
Deleting data is inherently risky. First, it’s essential to ensure that the data being deleted isn’t still in use. Second, data should never be deleted without creating a backup beforehand. Lastly, deletion might not be the best option at all, as the data could outlast the applications we currently use.
Atlassian didn’t use soft deletes and this led to a data loss. As they explain in their post-mortem, they didn’t verify if they were deleting the right records and had to recover from an incident. After the issue, they decided to always use soft deletes and not remove data anymore. All that could have been prevented. Let’s see how.
Database Guardrails to the Rescue
Development is rife with potential pitfalls, from slow queries and inefficient indexes to flawed architecture. While these issues must be avoided, traditional testing methods often fall short. To accelerate and improve development, we need to shift from conventional practices to full observability. Instead of merely validating query correctness through tests, we should monitor data changes, index usage, and execution plans. By leveraging telemetry signals, we can analyze query performance and predict their behavior in production, ensuring they won’t compromise the database or cause data loss.
Metis provides the solution to these challenges. It monitors all database activity across environments, including development and staging, capturing comprehensive details such as queries, indexes, execution plans, and statistics. Metis then projects these activities onto the production database, automatically evaluating their safety and potential impact. This streamlined feedback loop eliminates the need for developers to manually test their code. It also tracks all database interactions, including schema migrations and configuration changes.
For example, if Atlassian had implemented Metis, they would have received early warnings about potential data loss risks. Metis could have prevented the incident automatically, without requiring intervention from developers or database administrators. By delivering database reliability across all environments, CI/CD pipelines, and databases, Metis ensures safer and more efficient development workflows.
Use Database Guardrails and Stay Safe
Database observability emphasizes preventing issues proactively, progressing toward automated insights and resolutions, and integrating database-specific checks throughout every stage of development. Traditional tools and workflows are ill-equipped to handle the complexities of modern systems. Innovative solutions, such as database guardrails, tackle these challenges head-on. They enable developers to avoid inefficient code, evaluate schemas and configurations, and validate every step of the software development lifecycle directly within development pipelines.
Metis revolutionizes database management by automatically identifying and addressing issues. With Metis, your business can prevent data loss and database outages, ensuring seamless operations. Trust Metis to safeguard your databases, so you can focus on what matters most.
