Overview

Expectations are mental models of how someone thinks the software should work or deliver value. When someone’s experiences of the software are inconsistent with their expectations, cognitive dissonance can occur, which may lead to perceptions of low value. When someone’s expectations are met or exceeded, then this may lead to perceptions of high value.

People’s expectations are built up from ideas, previous experiences or artefacts called references, so called because they’re the points-of-reference for people’s expectations. These references are important in software testing because they give insight into people’s expectations and serve as oracles when evaluating quality. Any tester’s experiences or observations of the software that are inconsistent with one or more references could mean it goes against someone’s expectations and therefore may not deliver value.

For example, when a user sees a save button on an application, they will have the expectation (mental model) that clicking on it would commit any unsaved work on their computer to permanent storage. This expectation comes from reference sources such as:

• The save button working this way in other software applications
• The requirements, user stories or help files describing the function of the button
• The product owner or developers talking about how the save button works
• Other people saying this is how save buttons work
• The word “save” or diskette icon on the button means “to commit your work to permanent storage”

By building oracles from similar reference sources, testers can identify problems if their experiences are inconsistent with one or more references in a way that may cause lost value or harm.

 

References form the basis of expectations; someone’s mental models of the ideas of what software can do, should do or the value it can deliver. The same references also form the basis of oracles that testers construct to understand expectations.

When people use software, their experiences are compared against expectations, which form a crucial perception of value and quality. When testers use software, their experiences are compared against oracles, which allow them to evaluate value and quality.

 

Oracle Problem

It’s important to know that expectations can’t be fully known nor reconciled across everyone who has them, or there’s so many variables with anything other than the most trivial software that all possible expectations of what software should or shouldn’t do can never be fully considered without unreasonable cost. This means oracles are always partial and incomplete, and as a cognitive tool, they can’t be used to accurately test software:

A frequently invoked assumption in program testing is that there is an oracle. A program is non-testable if either an oracle does not exist or the tester must expend some extraordinary amount of time to determine whether or not the output is correct. The reasonableness of the oracle assumption is examined and the conclusion is reached that in many cases this is not a realistic assumption.

(Weyuker, 1982)

To even get close to fully specifying expectations in software, the below would have to be considered. This is unreasonable for most practical testing purposes so testers have to choose and hope they’re monitoring the relevant things they need to for testing (Hoffman, 1998):

• All inputs and outputs to the system at the time of the test, whether user or otherwise
• The entire system configuration state before and after the test
• All system data that’s changed before and after the test
• All system resource usage levels, such as fere memory, processor and hard drive space, before and after the test
• Any other software running on the system (memory resident programs), including their versions and states, and how they’ve changed before and after the states

 

Heuristic Oracles

Instead, oracles are heuristic: incomplete and fallible, but otherwise useful guides, in helping testers evaluate (make a preliminary judgement decision on) expectations and whether a problem exists. However, they are still open to false positives, false negatives and wrong questions, so understanding of the problem domain and skill of the tester is required to apply oracles effectively (Kaner, 2010).

Oracles are also used in algorithmic checks as part of the assertion that decides whether a check found a problem or not. Due to this nature, they can be machine programmed, but still suffer from the oracle problem in that the assertion is incomplete and fallible, so any check result still requires a testers’ judgement in deciding whether a problem may exist.

When applying heuristic oracles, the tester must consider:

• Something may not meet expectations yet still deliver value, for example, where expectations were low or missing to begin with
• Expectations are subjective, which means different people have different, or even contradictory, ideas about what the software can or should do, or how it delivers value to them
• Selecting and applying oracles requires understanding of the problem domain, skill and good judgement from the tester, to reduce the risk of false-reporting

When testing a software product, system, service or feature, a tester asks themselves or others:

• What are the expectations here? (Build oracles from references)
• How are they not being met? (What are the inconsistencies)
• Is there still a problem here? (Does it cause lost value or harm, or even extra value)?

 

References

A list of different reference types that can be used as oracles in software testing is available here.

 

Summary An important part of software testing is identifying references that form the basis of expectations, a mental model that someone has about the software. These references models are used in evaluating quality, as anything inconsistent with these reference models could point to a problem. Being able to describe and communicate these inconsistencies with reference models also allows effective reporting on quality to stakeholders by giving weight to the testers arguments and increases the reliability, validity and effectiveness of the software testing process.

Citations

• Hoffman, D., 1998. A Taxonomy for Test Oracles. Quality Week. Available at: Link p. 8
• Kaner, C., 2010. BBST Foundations 3A: Oracles. [online] TestingEducation. Available at: Link t.i. 1:32
• Weyuker, E., 1982. On Testing Non-Testable Programs. The Computer Journal, [online] 25(4), pp.465-470. Available at: Link

Updated 2023-01-11: Reworded and refined
Updated 2023-04-23: Minor retweak to oracles text