Scientific Method of Testing
These are adaptive changes from the more familiar scientific models (Crawford and Stucki, 1990). The later “analysis” phase has been replaced by “evaluate” to better match the qualitative nature of software quality. Analysis has then been recycled to a new phase spilt from background research to highlight a difference between learning about the product and understanding how it works by forming mental models (Hodder, 2018), These mental models then feed into the hypothesis. The testing scientific model forms part of the processes and activities of testing and is an important part of the story of quality.
Applied social research often starts with a question based on the problem that needs to be solved. For software, business stakeholders want to solve the problem of releasing unknown quality software. Testers provide information that solves a particular problem. Therefore information objectives guide the testing mission. Identifying the information objectives, what stakeholders actually want or need to know, is the first step. This broadly includes wanting to know if the product is ready for release, or whether it needs more development time (to improve quality) or testing time (more information on quality). With this in mind, the question that is asked of testers is:
“Are there problems that threaten the value of the product to people who matter?” (Bolton, 2020)
This extends to the quality of the project too, because anything that makes development, testing or business analysis harder or slower, affects the quality of the software.
Learn about the software under test. Perform background research into initial claims of quality, including what it’s supposed to do and what problem(s) it solves. Read documents and artefacts. Talk with business stakeholders and customers to learn about the problems, expectations, hopes, dreams and desires for the software. If any part of the software is available, investigate that too to learn what it is, what it does and what problem(s) it could solve, either directly or talking with developers. Use heuristic oracles to recognise problems when encountered, but also guide the necessary background research needed for effective testing.
“A good investigator actively searches for information, they don’t wait for it to be handed to them” (Kaner, 2020)
Seek out this background information wherever it may be.
Understand how the product, feature or service works or delivers value by breaking it down, forming mental models that feed into hypotheses. Quality is relative to someone’s subjective ideas, thoughts and feelings. Identify them here and use them to model out the software and problem-space with various risk, logistics, coverage maps and test design techniques.
Supposition new claims and stories of quality that can also act as charters to guide time-boxed testing sessions. Revisit existing hypotheses where evidence falsifies existing ones.
Interact and observe with the software using methods and tools to see what happens. The goal is to find evidence that falsifies claims and hypotheses value and to support (and falsify) counter-claims of threatened value. Feed this new information into analyses, hypotheses and evaluations.
Infer and conclude: decide whether hypothesise of quality are accurate based on logical reasoning supported or falsified by empirical evidence observed from experiments. Form an opinions on research question to give final suggestion to stakeholders.
Feedback and report findings to further guide testing and to communicate quality to stakeholders as sound stories of quality and threatened quality. Include a final recommendation on the research question.
There is no ordering to these phases, deductive and inductive reasoning alone has the hypothesis before and after experimentation respectively, nor should they be considered discreet or even mandatory, but a guide or heuristic to testing:
The scientific method is often misrepresented as a fixed sequence of steps, rather than being seen for what it truly is, a highly variable and creative process
A model for software testing that’s comparable to the scientific method reinforces that phases are entwined:
As an alternative to viewing testing as a series of discrete phases that only move forward (or back only with great pain and difficulty), we can instead consider each of these phases as threads that intertwine and influence one another throughout the entire testing lifecycle
Starting with background research as the entry point, each phase of testing Instead flows from, and feeds into, each other phase until the truth of quality is reached as much as deemed sufficient. Reporting to stakeholders in the actioning phase is then the final exit point.
This scientific model forms part of the overall quality research (or test) strategy, which is guided by the story of quality and any testing approaches.
- Bolton, M., 2020. TestingUy: Ask Me Anything – Michael Bolton. [Online] Available at: Link. t.i. 0:44:54
- Crawford, S. and Stucki, L., 1990. Peer review and the changing research record. J Am Soc Info Science, vol. 41, pp. 223–28
- Gauch, H.G., 2003. Scientific Method in Practice. Reprint Ed. Cambridge University Press. p. 3.
- Hodder, A., 2018. Phased Vs Threaded Testing. [online] Medium. Available at: Link
- Kaner, C., 2020. BBST® Foundations 3.0 Lecture 1 – Overview and Basic Definitions. [online]. BBST courses. Available at: Link t.i. 00:01:22