FAQ about software testing

when automation is worthwhile

Quality Assurance matters a good deal, if you aim for a quality product, of course. Let’s just deal with it. No matter how cool, experienced or skilled your developers are, their work is less effective without manual or automated testing (depending on the project requirements). When it comes to manual testing, everything seems clear: it’s a universal remedy used by a human against the majority of defects and put down in a series of test cases and reports.

Still, many misconceptions are associated with automated testing since its very early years. It has been around for several decades now, in this form or another. The commonly accepted and used meaning of ‘automated testing’ is using automation tools to execute the test case suit, unlike manual testing, performed by a human.

Some think it’s an end-all answer and problems reducer at early stages of a project development. Others treat automated testing as an extra to the budget and give it a miss.

Our Technical Lead of Automated Testing Department Stanislav Trubchanov will shed the light on the issue in the series of articles: ‘FAQ about software testing’.

What to automate?

At this stage, we will not dive deep into details, like how to test this or that function and such. To begin with, we simply list the cases where automation is recommended to be applied on the first place:

  • Hard-to-reach elements in the system. Some backend processes, file logging, and database entry.
  • Routine operations, such as data enumerating, better run when automated. For example, completing the forms with different data and a large number of input fields for further check.
  • Frequently used functionality, where risks of errors are quite high. By automating check of critical functionality, it is possible to guarantee rapid detection of errors, and hence, their rapid resolution.
  • Validation messages. Automate the filling of fields with incorrect data for testing form validation and error message appearing.
  • Checking the correctness of data retrieval
  • Long end-to-end scripts. Checking data that requires accurate mathematical calculations.

And this list is not complete. Depending on the requirements of the system under test and possibilities of a selected tool for testing it can go on and on. Moreover, for a more effective use of test automation, it is better to develop separate test cases that verify:

  • The basic operations of creating /reading /changing /deleting entities (the so-called CRUD operation - Create /Read /Update /Delete).

Example: creating, deleting, viewing and modifying user data.

  • Typical scenarios for using the application, or individual actions.

Example: a user visits an email website, flips through emails, scans new ones, creates and sends a message, leaves the site. This is the so-called end-to-end script, which tests a set of actions.

  • Interfaces, work with files and other moments which are inconvenient for manual testing.

Example: the system creates an xml file whose structure needs to be checked.

That particular functionality of the test automation can get the best output for your project.

The only way to go fast is to go well

Admit it, you can not save a project with a tight deadline by simply missing the tests. Writing unit tests (and it’s more than half of all tests as you can see below) is an integral part of any quality software development project. Otherwise, you have no way to find out if your code really works the way it should. Besides, you won’t be able to fix anything quickly, because you won’t know where exactly the bug is. So you better believe Uncle Bob, “the only way to go fast is to go well”.

We will list and describe the basic nuances of automation testing and give an answer to the question - when automation is worthwhile. To start off, it is necessary to bypass the shortcomings of automation and maximize its benefits. Advantages come first.

Advantages of test automation:

  • Repeatability - While working on a project some tests have to be performed multiple times. Automating the process can help minimize mistakes which are usually associated with monotonous and repeated tasks.
  • Smaller support costs. Indeed, it costs less if automatic scripts are already created, their support and results analysis usually take less time than doing the same amount of testing manually.
  • Fast execution - an automated script does not need to check with instructions and documentation, it saves a lot of execution time.
  • Reports on test results - automatically distributed and saved.
  • Performing without intervention - running automated tests can save an engineer’s time for other important tasks, or tests can be performed during off-work hours (this method is preferable, because the load on local networks is reduced during the night hours).

Some challenges of test automation:

  • Repeatability (yes, let’s see the negative side of it too) - all written tests will always be performed monotonously. A QA specialist when performing the test manually, can pay attention to details and, after several additional operations, find a bug. A script fails to do this.
  • The cost of support despite the fact that automated tests cost less than manual testing of the same functionality, the costs are still there. Additionally, the more often an application is changed, the higher these costs are.
  • Impressive development costs - development of automated tests is a complex process, because, in fact, an application is developed to test another app. In complex automated tests there are also frameworks, utilities, libraries, and so on. Naturally, all these need to be tested and debugged, and it takes pretty much time and money.
  • The cost of automation tools - for licensed software the cost can be quite high. Freely distributed tools usually have simpler functionality and less conveniences.
  • Missing minor errors - an automatic script can skip small errors it was not programmed to check. These may be inaccuracies in the positioning of windows, errors in inscriptions that are not checked, errors of controls and forms with which interaction is not performed while the script is running.

Now, knowing all the pros and cons of automated testing, you can make a decision whether this option is reasonable for our project. You just need to answer the question ‘Do the advantages of automating outweigh in this particular case’? If at least some of them do, you can go on. But if you struggle to find any, or drawbacks in your case are unacceptable — it is worth refraining from automation and stick to alternative manual testing.

Coming soon: ‘How to automate?’ and ‘What tools are the best to use?’ by our Technical Lead, Stanislav Trubchanov. To be continued.

We solve real-life challenges with innovative, tech-savvy solutions. https://chisw.com/