Tape-recorder — a tool for QA automation

Hi! My name is Victor Burov and I am a developer at ISPsystem. I would like to share my experience with QA automation.

Historically, the manual testing prevailed in our company for years and our testers spent a lot of time on repeating the same actions again and again. One day we thought: why don’t we teach a software to perform these actions instead of them. In fact, all of the actions turn into specific API calls in the end. This solution would allow people to write tests even if they don’t have programming skills.

We decided to write a module for creating automated tests. So that the tester should only press the button for creating a test, fulfill the test case conditions and click “finish” at the end — and that’s all, the test was ready! Such a simple idea, but it turned out pretty hard to implement. We wanted the module to highly-adapted to our products so we can an advantage of the unified interface and make the record look like a ready-made test case. It would completely eliminate the necessity of a manual work for writing the tests. We called this module a “tape-recorder”.

All the parameters of syscalls (like HTTP headers, environment variables, POST data, if any) and the entire response is written to an XML file. Every record receives sequence number. All the requests are divided into groups: modifying and non-modifying. After recording the test, many of non-modifying requests are deleted because they don’t affect the test execution and only slow down the execution process and make it more confusing and confuse the execution process.

During recording, the tape-recorder allows setting the check of values in the special form fields and in the list columns in just a single click. It also makes it possible to record negative tests remembering errors returned by the panel during a test recording.

During playback, the requests are sent directly to the application API without using a browser.

In fact, the tape recorder is a built-in module that allows us to install event-responsive handlers in all of our products. The module is built on a basis of COREmanager.

Unlike a human, a computer can’t “ just wait”. One of the first problems with the tape-recorder that we had to solve was writing waiters. We had to try many different solutions, but in the end we came up to implementing of background tasks waiting and adding the feature for stopping at a particular step for a specified number of seconds.

Do you remember those old school typewriters? One mistake and you need to retype the entire page.

We wanted to save our testers from rewriting the entire test in case of making a mistake. That is why we made it possible to edit the test from any step after saving it. This feature is very handy if you need to adapt the test to a changing of the behavior of a tested function.

When the tests were run, the values in the parameters being passed and checked varied depending on the server used for the test. A good example of such data is IP addresses. It is impossible to know them at the test recording stage, so I added the macros system. It allowed us to create tests that didn’t depend on the environment that much. Unfortunately, such solution requires a manual input of macros after recording.

Another problem that complicated working with the tape-recorder significantly was non — native keys. We haven’t noticed it at first, because the module was initially tested with ISPmanager, that used native identifiers. However, our other panels use a unique ID for identifying a record. That is why I had to teach the tape-recorder not only to receive the ID after creating a record or object (since the ID may change with every launch) but also to input it in all following requests.

The tests created by the recorder are run automatically in the Jenkins continuous integration environment. When tests are completed an xml-file containing JUnit data will be created. In order to provide a correct creating of the file, we applied a limitation for test naming. For example, the test User.Create.xml goes to the testsuite called User and its testcase is Create. In case of an error, a subsidiary failure node containing a full error description was added to it.

We count the number of unique functions being called during the tests and compare it with the total number of functions (excluding those available only for internal use) to find the percentage. It allows us to measure the simplest test coverage. In addition, the metrics show the total time of the test and the number of successful and failed tests.

The repository for tests was mainly designed for moving pre-made tests to other servers. It is especially handy when there are several QA engineers who write the tests. A simple panel for storing the tests was developed and deployed on the basis of our COREmanager. The tape-recorder has a built-in module for synchronizing the tests with the repository. When a new test is creating or uploading from the repository, it becomes unavailable for download in order to avoid a confusion. When a test is modified, the system increases its revision number to ensure that only the files being modified will be loaded into the repository.

When we started to use the tape-recorder it became obvious that some of the API functions didn’t meet our internal recommendations. In particular, some functions didn’t return the record ID after its creation. We had to go back to a working code and bring it into compliance with the requirements.

Deadlocks were another problem. The panel is intended to performing some critical actions in an exclusive mode. And since the tape recorder is a part of the same panel such functions may hang the entire system. We managed to determine it only after using GDB (no one remembered this feature at first). Unfortunately, the only solution to this problem was executing these functions in multithreaded mode. Theoretically, it was possible to design the tape recorder as a standalone panel instead of a module, but we haven’t tried.

The interfaces of our panels are changing with every release: it becomes more sophisticated and the number of elements is growing as well. That is why we constantly have to work on the tape-recorder to ensure it can analyze the structure of new elements and process request results.

Creating the tape-recorder allowed us to raise the quality of our main products. We saved a lot of time and resources of our QA team that we could spend on their education now. The tape-recorder helped us to check the compliance of our API and our own internal recommendations, making the API structure more logical.