The impact of automated testing on software quality assurance

By Vasiliy Lopatkin (vslopatkin@edu.hse.ru)

In recent decades, automated testing has become one of the key processes affecting software quality. Constantly increasing complexity of systems and growing performance requirements have prompted developers to look for ways to speed up and simplify testing, minimize the human factor and reduce the time required to verify software functionality and stability. Automated testing allows companies to detect defects early in development, preventing the accumulation of bugs and facilitating the release of more reliable products.

Automated testing encompasses a variety of methodologies aimed at quality control, and includes such types as unit testing, integration testing, functional testing, regression testing and load testing [1]. These approaches cover both individual components and the system as a whole, checking the compliance of the program with the specified requirements and stability to changing conditions. Each type of testing has its own peculiarities and is aimed at solving specific tasks: unit testing, for example, focuses on separate parts of the code, while load testing evaluates the program's compliance with the specified requirements and stability under changing conditions. while load testing evaluates the system performance under high loads.

Various specialists deal with the implementation of automated testing in companies. For example, quality engineers (QA) develop and execute automated test scripts to verify that the code works. Developers often write tests for their modules to make sure they are correct before integrating them into the main project. DevOps experts often develop and maintain infrastructure for automated testing, including CI/CD pipelines, which allows tests to be executed automatically whenever the code is changed. Collectively, this allows for a higher level of quality at every stage of development.

Thus, automated testing has become an integral part of software development, and its impact on product quality cannot be overemphasized. This essay examines how test automation affects the quality of the software by avoiding common errors and speeding up the product release process.

An important aspect that affects product quality is test automation. Ramler et al. describe the benefits of automated testing, which helps reduce the cost of re-testing and improve quality by reducing the number of bugs in releases [2]. Another study emphasizes that test automation helps to quality improvement by providing fast and accurate defect detection, which reduces the number of bugs that go into the final product. The authors point out that the use of automated tools speeds up the testing process, allowing developers to adapt tests to specific scenarios and effectively maintain quality at all stages of the software lifecycle [3]. Studies by Kasurinen et al. confirm that automation is widely applied for quality assurance and quality control in industry, although the implementation of automation requires significant effort [4]. Damm et al. describe how automation in development-oriented testing, facilitates early detection of defects, which also has a positive impact on product quality [5].

Numerous studies [2, 3, 4, 5] confirm that automated testing significantly improves software quality, reducing the number of defects and speeding up testing processes. For example, studies show that the use of unit tests, developed at the initial stages can reduce the number of errors detected at later stages, as well as reduce the cost of fixing them. Studies on integration and regression testing point to their usefulness in continuous integration as they allow you to monitor the stability of the product with each change. An important aspect here is also the fact test automation promotes standardization of processes and simplifies test documentation, which makes testing reproducible and scalable.

Despite the recognized advantages, test automation also has limitations. Some studies draw attention to the high costs of implementing automated testing, especially in the initial stages. Automation requires time and resources to develop and maintain tests, which is especially critical for small companies or startups. It is also emphasized that automated testing is not always effective for complex user scenarios and may be inferior to manual testing in situations that require complex interaction scenarios.

With the advent of new technologies such as artificial intelligence (AI) and machine learning, test automation is also undergoing changes. Many modern AI/ML-based tools promise to significantly improve testing efficiency, but some of these claims are exaggerated and need to be critically analyzed [6]. and need to be critically analyzed [6]. For example, using AI to identify interface elements and prioritize tests helps to automate routine tasks, but for complex contexts such methods are not yet always successful. It is important to choose an appropriate solution depending on the role of testing in the company - quality control (QC) or quality assurance (QA). This direction, however, is not yet sufficiently explored, and most of such solutions remain at the experimental stage. The question about the long-term effectiveness of such approaches and their application in real commercial projects also remains open. in real commercial projects.

Although the benefits of automated testing for software quality are widely recognized, some aspects are still poorly understood. For example, there is limited data on how test automation affects complex, multi-component systems, in which multiple services and microservices interact with each other. In addition, it is important to further investigate the ROI of of automation at different stages of the software lifecycle. Research in this area will provide a better understanding of, how to optimally allocate testing resources and when automation is truly beneficial.

At the moment I have a combined three years of experience working on two large commercial projects, that had very different approaches to automated testing.

1 - Sigma, the revenue calculation system of S7 Airlines. This is a microservice application, the main task of which was to process all incoming transactions for the previous day. It used an approach in which the main automated tests were E2E tests. Unit tests and integration tests covered only 30% of the code (only the core of the system), which significantly accelerated the development of new features, refactoring of the legacy code, and E2E allowed to refactor the legacy code. code, and E2E allowed for fast regression testing. This approach allowed us to speed up the development of new features significantly, while at the same time the number of Customer Bugs was no higher than in projects with more than 70% of the code covered by unit tests, and automatic launch of regression testing when building a release allowed not to keep more than three QA and regression testing engineers in the team.

2 - Sber Insurance. This is still a microservice application based on the tasks of insurance policy issuance, loss settlement and post-sale service. service. In this project the main part of testing automation was in unit testing, while regression testing was performed manually for 90% of the time. testing was performed manually by 90%, because of which the QA testing team consists of 8-10 people. At the same time the percentage of Customer Bug is quite high. The most likely reason for this is the human factor. In case of Unit tests it is insufficient qualification of the developer in the testing section, in the case of QA it can be banal fatigue or deadlines that prevent from testing completely or more focused.

Analyzing the peculiarities of both projects I made a conclusion that it is not so important how many tests there are, how many varieties of tests they are, but rather it is important to get away from the human factor in manual and unit testing. Automation should be handled by people with competences specific to this area, not developers or QA engineers.

In conclusion, automated testing has shown significant potential in improving software quality by minimizing manual intervention, reducing defect rates, and speeding up release cycles. Through the review of methodologies and my own experience on commercial projects, it becomes clear that the impact of automation is maximized when managed by specialized professionals, rather than being added as an afterthought by developers or general QA engineers. While automation can streamline regression and quality control, a balanced approach that minimizes human error at all testing stages is crucial to achieving optimal results in software projects.

1. Автоматизация тестирования против ручного тестирования: Заменит ли автоматизация ручных QA специалистов? https://habr.com/ru/articles/716402/
2. Ramler, R. & Wolfmaier, K. Economic perspectives in test automation: balancing automated and manual testing with opportunity cost. https://www.irisa.fr/lande/lande/icse-proceedings/ast/p85.pdf
3. Hussein Mohammed Ali, Mahmood Yashar Hamza, Tarik Ahmed Rashid A Comprehensive Study on Automated Testing with the Software Lifecycle https://arxiv.org/pdf/2405.01608
4. Kasurinen, J., Taipale, O. & Smolander, K. Software test automation in practice: empirical observations https://onlinelibrary.wiley.com/doi/10.1155/2010/620836
5. Introducing Test Automation and Test-DrivenDevelopment: An Experience Report https://www.researchgate.net/publication/223215802_Introducing_Test_Automation_and_Test-Driven_Development_An_Experience_Report
6. AI/ML в автоматизации тестирования программного обеспечения https://habr.com/ru/articles/648621/