We take CI/CD for granted nowadays. We test our application with unit tests or load tests before the deployment, we store immutable artifacts, and we deploy changes automatically. And yet, things may break in production, we may need to roll back changes, and we may still face unexpected outages. How is that possible?
Are you sure that your CI/CD covers everything? While it may sound obvious, there are so many things that we miss. We don’t check our queries properly, we don’t control configurations, and we don’t cover non-functional requirements in general. Let’s see what 5 things we lack in our CI/CD pipeline and why they are crucial in today’s world.
First, are queries fast enough?
When developers test their solutions, they focus on their correctness. They verify if they read and write correct records, whether they return proper results from their APIs, and whether they don’t lose any records during writes. However, they don’t check if the queries are fast enough.
There are many reasons why a query may be slow. It may use inefficient techniques (like CTEs), may extract too much data that is not used later on, or the database may store the data inefficiently (like in strings instead of binary structures).
Even if we run load tests, they happen very late in the pipeline and are expensive to write, execute, and maintain. We need something better.
Recommended reading: How Metis Enables Teams to Test Their Databases at Scale
The solution is to verify the execution plans right when the developers are implementing their changes. We can take their queries, project them on the production database, and verify if the query is going to run fast enough. This is something that we should do as part of our CI/CD pipeline automatically. Metis helps with that by integrating with your applications and databases, extracting statistics, schemas, and configurations, and then verifying if all is going to work well after the deployment.
Second, are migrations fast enough?
Migrations can take your databases down for minutes or even hours. The database engine may need to rewrite the table. In such a case, the database copies the table on the side, modifies the schema of the table, and then reinserts the data again. The table may be offline during that time which blocks the whole database and application.
We don’t verify if migrations are fast enough. We run them outside of our test suites and therefore the tests come to the state when the migrations are already done. We don’t verify how long those migrations take. Not to mention that they typically are fast because our tests run against small databases. Even worse if we don’t use real databases for our tests but go with in-memory replacements or mocked solutions. In these cases, we don’t need to run migrations at all as we just create the database in the target state.
We need to verify migrations as part of CI/CD pipelines. We can similarly do that as verifying queries. We just take the migration code, extract the execution plan, and verify if it’s going to run fast enough in production. Metis does that thanks to its module dedicated to migrations and CI/CD integration.
Recommended reading: Common Challenges in Schema Migration & How To Overcome Them
Third, are indexes used?
When queries are slow, we typically add indexes. They are great and can improve reading performance significantly. However, they may also impact the performance of writing the data as the database needs to keep the indexes in sync.
We typically don’t focus on indexes and if they are used correctly. Even if we check if our queries are fast enough, we don’t check if they use indexes per se. This often leads to a state when we have too many indexes which are fast enough but unnecessarily slow our databases down.
We need to verify which indexes are used and how. We need to track unused indexes and remove them to improve performance. Metis tracks indexes and alerts you when they go stale or unused.
Fourth, does the production have the same configuration as non-production?
We test things against local databases. We may inadvertently rely on specific configurations like extensions, buffers, caches, enabled modules, or statistics. But how do we know if the production database is configured in the same way?
To verify that, we need to extract configurations from both databases and simply compare them. This is something we don’t do as part of CI/CD. If we apply GitOps, we may have the configuration deployed automatically each time. However, this still doesn’t guarantee that the configuration is the same. Even worse, it’s most likely different. We scale our production servers differently than our non-production ones. We need to understand how the configuration affects performance. Metis supports that by comparing the net effect of configuration settings between production and non-production systems.
Fifth, is the current state expected?
Last but not least, our CI/CD pipelines modify our environments and only verify them at a point in time. They don’t track if the changes in the environments are expected.
Let’s take a query that uses an index. The index was added to improve the query performance. We deployed the changes and we verified that all is correct and fast enough. Later on, we changed the schema so the query doesn’t need the index anymore. We deploy the changes and verify all is good and fast enough. CI/CD pipelines are green and we may think all is great.
However, the state in which we arrived was not the state that we wanted. We have fast queries and indexes, and all tests are green. But we don’t need the index anymore. This is something we can’t verify in our CI/CD. The pipeline only checks if the query is fast, but the pipeline doesn’t verify if the reason for adding an index still holds.
Metis helps you with that by tracking all the changes and explaining how they affect the production database. You don’t rely on the sheer fact that the queries are fast. You also get an explanation of why they are fast enough.
Summary
Our CI/CD pipelines are great but they don’t cover many aspects of our databases and environments. We can’t let that happen. We need to verify if our queries are fast enough, schema migrations won’t take the database down, indexes are used properly, the configurations are the same, and the state we are at is the one we expected. Use Metis to fix all these aspects and make your CI/CD pipelines rock solid!
