Load tests are just a waste of time. Let’s read on to understand why.
Problems With Load Tests
Load tests should answer if the service and the database are fast enough to be deployed to production. They do this job well but have so many drawbacks that ultimately they should be avoided.
First, load tests take too much time. We must run thousands of transactions for many hours to get meaningful results. We need to fill caches, drives, and networks, and keep it running for a long time. Effectively, we can’t get results early. We typically let the tests run overnight or just trigger them in the CI/CD pipeline and don’t wait for their results.
Second, load tests are hard to write. We can’t simply shoot the service with random requests. We need to mimic the production data distribution and send the data that just makes sense. We need to deal with stateful services, prepare databases, and run things that cover all the code paths in the application. This is challenging, especially when the service behaves differently in different countries or continents, and when it evolves quickly over time. Getting meaningful results is non-trivial.
Third, load tests are difficult to maintain. Data changes over time, services change their implementation, and we need to keep our testing data up to date. We may want to simply replay the production traffic but then we risk that we’ll have an invalid state and not hit the right code paths.
Fourth, we need to deal with GDPR, CCPA, and similar. We can’t simply take the production data and play it in the non-production environment. This often breaks the security policy and is very dangerous. We need to anonymize the data, remove social security numbers, and make sure customers’ data do not leak. This is not easy and poses many risks.
Fifth, randomness. It is perfectly fine to get a different response for the same request due to randomness, variations in the timestamps, random identifiers, and other stuff. This makes writing assertions much harder as we can’t simply compare things byte by byte. We need to understand the structures and compare their semantics.
Sixth, they happen far too late. Even if we find issues with load tests, it’s already after we wrote the code, reviewed it, merged it, and deployed it to some environments. If we now find the issue, then fixing it is very expensive, as we basically need to go back to the whiteboard. This is a tremendous effort.
There are more reasons why load tests are inefficient. However, what should we do instead? Read on to understand.
Observability Is The Key
Instead of running load tests, we should catch the issues early. We can employ many observability techniques to see things behind the scenes and identify the issues in the developers’ environments.
We can use telemetry to capture the queries sent to the database. Once we have these queries, we can project them onto the production database and extract the execution plans. This way we can immediately tell if the queries are going to be fast enough when deployed to production.
In the same way, we can verify schemas and configurations. We can check if the indexes are configured properly, and if the queries use indexes well. We can verify what data is read and if there are better ways to do so. Most importantly, we can do that automatically and compare it with the production database. This way, developers can get immediate feedback right in their environments. They don’t need to submit code reviews or deploy to staging environments.
We can utilize observability to do much more. Since we know what is being changed and in what way, we can easily pinpoint issues in production and correlate them to the actual code changes. This way, we can submit automated pull requests that fix the issues. We can tune configurations, schemas, indexes, extensions, and much more. This way, we achieve database reliability and automated self-healing.
Metis Helps You Save Time and Money
Metis analyzes the queries right in the developers’ environments:
In the same way, Metis analyzes schema migrations and protects your databases from performance degradation and data loss:
Metis keeps your whole database under control:
Summary
Load tests happen way too late in the pipeline to be effective. They are very expensive, hard to build and maintain, and take much time to execute, and they happen after we already merged and reviewed the code. We can use observability to avoid these issues and identify problems much earlier in the pipeline. Metis handles all of that automatically and helps us achieve great database reliability.
