References are the ideas and artefacts that form the basis of someone’s expectations in software. Expectations are a type of mental model consisting of preconceived ideas about what the software can do or the value it delivers. Testers seek out references to use as oracles when testing, as anything inconsistent with one or more references could be something that goes against someone’s expectations, and therefore threatens value to them.

References are also useful in test design as identifying references provides guidance on where the tester can look for the problems that matter. References are also useful in bug advocacy, as testers are able to describe any problems uncovered to stakeholders using inconsistency with one or more references giving weight to their logical argument and allowing better business decisions (Bolton, 2005) (Bolton, 2012)

There are many different references that can be used as oracles, however as heuristics, these oracles should be read with “potentially” or “reasonably” and not a hard rule to be applied in all circumstances, but are otherwise useful, powerful tool. There’s no limit to the number of references or reference types that exist, or how they can be described or grouped together. Below is a possible list of reference types to be used in testing inspired FEW HICCUPPS (Bolton, 2012):

List

• Intention Consistency with the intended purpose of software and the value it delivers to stakeholders (whether implicit or explicit)
• Documentation Consistency with explicit claims made in project documentation, such as specifications, guides and notes
• Conversation Consistency with explicit claims mentioned by stakeholders in meetings, discussions, stakeholder interviews or in-passing
• Application Consistency with other software products, systems or services that share similarities in purpose or function
• Version Consistency with past versions of the same software program
• Reflection Consistency with the software program itself, whether the same feature or function, or similar features and functions
• Jurisdiction Consistency with national or regional governmental, legislative or judicial regulations, statutes or precedents
• Standardisation Consistency with recommended best practices by external professional bodies and recognised industry experts
• Organisation Consistency with policies, procedures and precedents set internally by the organisation
• Reputation Consistency with how stakeholders wish to be seen or the image they wish to project
• Restriction Consistency with real-world constraints, such as language, mathematics and data
• Problem Inconsistency with previous problems that stakeholders decided needed fixing (which relate to another reference above)

Citations

• Bolton, M., 2005. Testing Without a Map [online] Better Software Magazine 2005-01, pp.25-28. Available at: Link Alt
• Bolton, M. 2012. FEW HICCUPPS [online] Developsense Blog. Available at: Link

Update 2022-09-15: Rewording and additional citations