Introduction

Like many activities and professions, software can be used to help software testers solve problems, complete tasks or achieve goals by reducing the cost of certain tasks to allow the running of a test or collection of tests that would otherwise be impractical or impossible. Automation tools for testing are available off-the-shelf (ready made) or created internally by the software project team (“tool-smithing”), or a mix of both, and is also known as computer-assisted testing (CAT) or Automation in Testing (AiT) (Bradshaw & Winteringham, 2018).

In summary, automation in the context of software testing can be defined as:

• A software project or sub-project to manage, develop, test, deliver, maintain and operate software to support testing of other software
• The software tool, product, system or service produced as an output of the above software project

Anyone with a project team can be involved with automation in testing, or responsibility for it maybe shared among two or more roles within a team, where it typically forms assessment criteria for definitions of done in agile (Jones, 2018).

Automation in testing can fall into two categories, support tools and automated checking, where the focus in this section will be on the latter, writing and executing encoded checks.

 

Support Tools

Support tools refer to anything that helps a software engineering team test but isn’t automating part of the testing itself. Examples include:

• Note-taking (mind-mapping, word-processing etc) for collecting and collating test ideas or results, and for reporting
• Communication software (email, instant messaging, video calling and conferencing)
• Project management software (task trackers, bug trackers),
• Screenshot and screen recorders
• Image editing software
• API tools for sending requests and receiving responses
• Integrated development environments (IDEs) for developing test automation tools

 

Automated Checks

Automated checking refers to writing and executing of coded checks or test cases where the algorithmic nature of checks or test cases make them suitable for programming. The basic, shallow nature of automated test cases allows fast, broad feedback on any unexpected outputs helping the project team quickly (with low cost) detect where the program isn’t doing what the they intended it to do, whether for newly added value or regression of pre-existing value. This can be especially beneficial when combined with automated delivery as it enables continuous integration, delivery or deployment (CI/CD) pipelines that provide feedback right after developer code check-in (Jones, 2018).

Automated checking can be considered non-interactive, instrumented testing and may be performed as part of a wider test (Bach & Bolton, 2021).

Automated checking may not include all aspects of a test including the evaluation (assertion) instead focussing on one particular area of the test only. For example, automated checks  may be employed with exploratory testing (known as automation in exploratory testing) to uncover new problems and risk using other reference oracles. An example is assisting with tests that aim to uncover problems through mass data processing and pattern recognition, where the data is automatically generated, collated and/or displayed. Another example is the automated communication of test results.

 

Caveats and Considerations

Automation Bias is the complacency and overreliance on automated tools and systems, including the assumption the automated tool itself is infallible and problem-free. However automation can only be used to support human activities by reducing the cost of some aspect of it and cannot replace the activity completely. A common misconception among project teams and management that automation in testing can replace all testing activities in a software project, but that’s only partly true if the testing mission is fast, broad code output checks, otherwise opens up huge scope risk in areas the automation tool can’t uncover. Even then the quality of the automated test, as well as the output, must be scrutinised and evaluated respectfully for mistakes in the automation and whether an actual bug exists (a failed automated check doesn’t automatically mean there’s a bug in the software, it could be a bug in the automation tool).

Oracle Problem is the recognition that oracles are incomplete, fallible guides in software testing which means basic automated assertions with a true or false binary output have a particularly high chance of false positives and negatives despite the fact they appear objective and absolute misleading stakeholders into thinking they are.

Anchoring and bandwagon is the high tendency to focus on one type of oracle and one type of risk in automated checking, reference versions for regression risk, at the expense of a more context aware analysis of risk and reference models in a software project. Due to false positives and negatives, the second problem is most problems found are with the automated check itself, requiring constant costly maintenance without much valuable information (Kaner, 2010b).

 

List of Topics

• Elements: Setup (data and configuration), Action (performing interaction, Assertion (expected vs actual output result), Reporting (logging), Clean-up
• Pyramid: Unit vs Integration vs End-to-End (GUI, API) vs Layer (GUI, API, DB)
• E2E: Using test automation tools to perform “thin slice” checks through the stack (typically GUI-to-DB or API-to-DB)
• CI/CD: Using automated tests to enable automated integration and deployment pipelines
• Testability: (including hooks)
• SVD (Self-Verifying Data)
• TDD/BDD/SBE: Test-driven development, behaviour-driven development, specification-by-example

 

Citations & Further Reading

• Bach, J. & Bolton, M., 2021. Alternatives to “Manual Testing”: Experiential, Interactive, Exploratory. DevelopSense. Link
• Bradshaw, R & Winteringham, M. 2018. Welcome to Automation in Testing. Automation in Testing. Link
• Kaner, C., 2010. BBST Foundations 3A: Oracles [online] TestingEducation (BBST). (a) Link t.i. 6:53, (b) Link t.i. 1:26
• Jones, A., 2018. Chapter 1: Designing a Test Automation Strategy. Test Automation U: Applitools. Link