Milos Najdanovic, QA Engineer at HTEC
Depicting a QA as a mere software tester whose job is only to validate the functionality of the application manually is a thing of the past. As companies transform and transition their businesses into the digital arena, and the applications become increasingly complex, the role of a QA is to ensure that all parts of the system are functioning properly – the response times are fast, the security is airtight, and user experience is in line with modern standards. A lot of these new systems have complex logic on the server-side and testing them manually is at times not possible, but even if it were, it would be time-consuming and highly unreliable. Nevertheless, manual testing is still a big and an important part of a QA’s job, even though, on a larger scale and in the long run, most parts of the product must be covered by automated tests. Test organization is most commonly graphically depicted as a pyramid.
Test Pyramid
The first step of test automation is on the unit level, where a unit is considered the smallest testable part of the application. Unit tests are usually written by software developers, and if they do not pass, the application cannot be built. However, this is only the first line of defense, making sure that the application functions on the basic level before it reaches the tests on the service level.
The service level, which usually covers backend services and business logic, should lie within the responsibility of QA Engineers. One viewpoint is that integration, functional and non-functional tests can be performed at this level.
End-to-end UI tests and manual tests, which cover UX and edge cases, take up a smaller portion of the testing process. Manual testing, by itself, is seen as lying at the end of the test cycle, although it actually performed throughout the entire process of development.
As HTEC’s QA Engineer Sandra Jeremenkovic likes to note, “test automation is like airport security – in order to board a plane, you have to go through different security checkpoints and get rid of the excess baggage and all that is prohibited from going into the air.”
Integration Tests
What Integration Testing implies, differs from one company to another. In HTEC, Integration Tests are performed by Quality Assurance Engineers; they cover API and work with only GET endpoints and POST methods that do not change the state of the app or database. This includes checking the structure of the response, reviewing all properties, data types, required fields, and status codes and going through responses with all the application roles. These types of tests are implemented quickly, they are easy to maintain, and their execution is fast. They can be run in parallel and should be a part of the Continuous Integration system. Integration Tests should ensure that when a new app version is deployed, the rules for different roles still apply (thus ensuring the app security on the API level) and that the structure of data and data types is not changed and will not affect the end-users with previous app versions.
Functional Tests
Functional Tests should cover the business logic of the application and are slower to integrate and execute than Integration Tests. They aim to cover the features that are hard or impossible to test manually, as well as to lower the time needed to do the regression testing manually. These types of tests take a lot of time to integrate and usually require a dedicated QA Engineer, but they are the core of test automation and give the biggest benefit to the quality of the product. In terms of Continuous Integration, the Integration Tests can be set to be triggered on deploy and can roll back the changes if they fail. On the other hand, Functional Tests should run on schedule or after the deployment, depending on what kind of CI is set.
Non-functional Tests
Integration and Functional Tests cover the response structure and the app logic, while Non-functional tests should focus on the actual performance of the application. Non-functional tests check how much time the servers need to respond, how much load the app can take, and at which point the application could break. All of them can quickly be implemented and executed. However, stress tests are an exception – they should not be run too often, and they should be performed only at times when not many users are active. If performed on AWS, they need to be approved by their admins so they would not be flagged as a DDOS attack. By comparing reports, QA Engineers can analyze how response times change from build to build and accordingly schedule code refactoring and improvements. On larger systems, Non-functional Tests can also check if the server auto-scaling triggers or prevents the app downtime. These types of tests, in general, rely on the collaboration between the DevOps and the QA Engineers for server-side improvements.
End-to-end Tests
Although in some other systems priorities may be different, and end-to-end tests may have higher coverage, tests related to User Interface are often kept at the top of the testing pyramid. With certain test tools, QAs can map the UI elements and interact with them, and their goal is to simulate the end-user behavior. They are slower to implement, slow to execute and can at times be hard to maintain. The apps which go through frequent visual changes and improvements make it harder for QAs to catch up. In order to reduce the time needed for manual regression testing, it is necessary to cover at least the basic screens and the flow of actions.
Manual Testing
If the application is covered with all the aforementioned automated tests, manual testing is focused around exploratory tests and searching for corner cases. The designated QA will, in this case, have time to cover User Experience, suggest visual improvements and do the testing on different operating systems, browsers, devices, etc. Manual testing is, therefore, crucial and cannot be fully replaced, but with good automated tests the coverage time can be saved, and the chances for bugs minimized. Moreover, automated tests are the only way to properly guarantee the quality of the developed applications, especially those with complex logic, high security and a lot of users.
Conclusion
We created a small application with not that complex logic (React on the frontend, NodeJS on the backend) and ran different types of test on it. As seen in the graph above, the fastest to execute are Integration and Performance Tests. Moreover, in the time frame of around 25 minutes, they can be used to cover all of the major parts of the application. High test coverage and its implementation into CI can also guarantee faster and more frequent deliveries to the production environment while maintaining high quality. In the graph below, we can notice that by ramping up the number of executed test, the overall completion time will still not be that high, with the exception of UI tests. As mentioned before and considering the given metrics, it becomes obvious why UI tests are on the top of the pyramid and why they should cover only the basic flow.
For huge systems, not investing in test automation can have serious repercussions. The bigger the system, the more manual software testers it requires. However, the risk of human error is pretty high and so is the risk of bugs, app downtime or security breach. Big businesses that store sensitive user data are frequently targeted by hackers and any data leak can lead to stock value plummet, lawsuits or can potentially drive away users. In the era in which people are used to relying on mobile and online banking systems if a bank cannot guarantee the app security, the users will avoid its services because they do not want to gamble with their life savings.
Even if a business is in the startup phase or preparing an MVP, or if they depend on external investments and funds, they should definitely dedicate their time and resources to test automation. In the long run, test automation is a sound investment. Once the business starts to expand and the user database is on the rise, implementing test automation may take much more time, human resources and money. QA Engineers, at this point, will need to focus on two fronts – covering the current app and syncing with the development team to cover all new features that are in the works. New features will keep rolling out, the pre-existing logic will still not be covered, and the project will require onboarding of the new people to do the manual testing to keep up with new deliveries.
Quality Assurance as a profession is relatively young, especially in Europe, and QA test automation is even younger and seen as an uncharted territory by most businesses. There is no right or wrong answer when it comes to choosing the proper test strategy because they tend to differ from one project to another and should be adjusted accordingly. Therefore, our test pyramid is just one possible way of approaching the automation and not a universal template. Regardless of the chosen organization of tests, dedicating time and resources to test automation is essential for any business which seriously prioritizes quality.
